AI Generative Market Data Protocol Decoding - Part 1

Written By Aaron Foo

Its finally here, AI has fundamentally changed software development forever. Having dabbled in it over the last 2 years, from watching it fail the most basic coding task 18 months ago to its current point in 2026 July with the release of Fable 5, the pace is incredible.

While AI Code can do alot of things, theres certain areas where it fits extremely well. One of those is network observability because;

  • Its NOT in the critical path

  • Instructions and specifications are clear

  • The output is verifiable using a 3rd party

Its not generating ultra low latency trading decision logic, or writing a nuclear reactor control process. Its using these tools to improve network the visibility, typically going from little to no viability. Or from an extremely expensive solution to something a fraction of the cost.

Put these things together and FMADIO x AI Decoders and you have a bespoke network analysis toolkit tailored precisely to your environment for pennies on the dollar.

Disclaimer: Written by a Human, apologies in advance for the poor grammar.

AI Development Model

While you could just load up Claude and say the following

Write a market data protocol decoder for NASDAQ ITCH5 and output it in Parquet format

Which it will happily try do, it might get close but what you really need is a development mindset and framework as the AI’s have a tendency to wander off in odd directions, not unlike a team of human engineers. Creating structure and management of the project from the start is critical, this ensures its on a solid foundation and can be rapidly scaled without compromising quality.

We are using Anthropic Claude Fable 5 for this (July 2026), running entirely on the Linux command line with Claude Code in an Ubuntu 22 LXC container.

Shown below is how to think about development with a prompt. The goal is to ensure the Prompt gets into a feedback loop, grounding it using real verified data. This enables it to generate code, run an experiment, check the results of the experiment, review the results, modify the code again and so on. In the old world this would be called reinforced supervised learning, e.g. training with human tagged data, or the human equivalent would be TDD, Test Driven Development.

In the modern AI Code world it’s called…. I don’t know what. Things are moving so fast, there is probably some new word for this, I’m calling it slopping the vibe on the ground.

Conceptual AI Development Model

Grounding with Reference Data

For market data protocol decoders because the inputs (PCAPs) are clear and standardized. The network protocol specifications are all static and well defined, this creates an almost complete feedback loop, the only missing part of this is the Reference data so the Model knows how to ground itself.

Reference data is somewhat like the “Null Hypothesis”, e.g proving the opposite result is impossible, thus the result presented is correct. In this case we are proving the protocol decoder is working correctly, because there is no difference between the reference (correctly decoded) data and the data the decoder generates.

This is a key point, as it makes a huge difference to the Model’s ability to reason by itself, allows it self correct and fix its own bugs without your involvement. Which in the grand scheme of things is half of what the software development process is… testing, fixing bugs, repeat.

Specification

Next is the specification, giving the Model full details of what it has to do, how it has to interpret data is an absolute critical part of the process. The key point is:

Specifications the Model can easily understand and reference

The above is a crucial, the language of AI Models is markdown, the best way to do this is take the raw specifications in PDF, Text, XML, Web pages or whatever and using the AI itself to generate the specification document. Getting the model to write this specification itself is half the battle. For market data it’s quite trivial as its a highly detailed and precise network protocol specification, where every bit and byte is clearly defined.

In addition to the network protocol specification this application must automatically detect what protocol it is based entirely on the PCAP. To do this we need each venues Network address / subnets / vlans / multicast group in addition to the wire format in the specification document.

Project Structure

The project is structured into the following components:

  • Core Packet Loop

  • Output Writing Format (in this case Parquet)

  • Per Protocol Specific

    • Specification

    • Decoder

    • Verification

The GItHub project is shown below.

FMADIO Markets GitHub Market Data Protocol decoder project

Core Packet Loop

The core packet loop is quite basic, its a standard PCAP packet loop as shown below, with a bit of status prints to show its progress thru the PCAP. The first prompt used to generate this is shown below.

start by creating a main.c that reads in standard PCAPs with nanosecond timestamping. it sends each packet to a ProcessPacket() function which does basic VLAN stripping and TCP vs UDP identification. TCP and UDP go to a ProcessTCP() and ProcessUDP() function.

Toplevel Packet Loop

The ProcessPacket function does the usual VLAN, IPv4, UDP decode sort select etc, resulting in a ProcessUDP() packet loop as shown below

Above is where all the protocols get detected and decoded, this will end up becoming quite a large function. This will be flow based in the future, to reduce the work protocol detection has to do for each packet, for now will keep it simple as above.

NASDAQ TotalView ITCH v5

Next is the specification, as mentioned above we use the Prompt to generate the nasdaq_md_itch5.md file. The source files are the PDFs.

All of these PDFs are checked into the GIT repo in the per protocol directory as shown below:

Source documents for the AI to read

And then we issue our prompt to generate the specification file.

now read the pdf protocol specification files in protocol/nasdaq_md_itch5 and generate a .md file which summarizes all the information requried to decode this protocol

Which generated the ITCH v5 Speficiation.md file. The first section atleast, additional information is added in later prompts.

Prompt to generate the specifications MD from various raw source data.

Next is include the multicast addresses and network information to detect what protocol a packets is to be decoded with, the prompt for that is below

read the file UDP_IP_Address_US_Equity_Market_Data_Feeds.pdf and update the .md file with all the subnet and port and multicast information

And we have the specification document in a format the AI (and human) is optimized for. One of the key points is we can go back to modify this reference document and iterate on that, then feed it to the AI code gen pipeline without blowing up the output, in other words it keeps the AI focused.

The Schema

Next is creating the Parquet schema. This goes into the specification document so we can tweek the specifications manually (or via the AI) and have it update the generated decoder. The prompt to add the schema is below:

using libduckdb parquet generate a schema for the UDP protocol decoder in protocol/nasdaq_md_itch5. this schema should be a flat schema and include every column in the protocol. write the schema out into the same nasdaq_md_itch5.md file in a new section

And like magic is goes ahead and reduces all the fields into a flat schema, noting the duplicate use of fields such as Price.

Prompt to generate the ITCH Market Data Protocol Schema.

The specification schema is linked below:

https://github.com/fmadio/fmadio_markets/blob/main/protocol/nasdaq_md_itch5/nasdaq_md_itch5.md#7-flat-parquet-schema-duckdb--libduckdb

Generated Flat ITCH Schema, nice how it identifies duplicate fields

The code version below:

DuckDB Schema for generating the PARQ file

The Decoder

After all this preparation we finally get around to generating the decoder. Its important to build the structure and specifications first, as it allows us to scale and add additional protocols easily. All thats required now is the prompt below.

using the specification in protocols/nasdaq_md_itch5/nasdaq_md_itch5.md generate a C99 protocol decoder that outputs a parquet file using libduckdb

And like magic is goes ahead and generates the decoder file nasdaq_md_itch5.c taking all of 6min to do that. A human to writing this code would take hours if not days to fill out every single message.

… and just to humble us some more, it decided write code by itself to synthetically generate ITCH PCAP messages for testing.

Claude being proactive on the test generation.

It’s now completed the generation, reports what its done. Fixed a few of its own bugs using the synthetic data it generated. Brilliant.

Claude Finishes generating and Verifying the ITCH Decoder.

Of course there is more to it than just these prompts, it requires a bit more glue to prompting to connect it all together. Yet the amount of additional effort required is minimal, orders of magnitude faster than manually writing the code.

And finally, below is the output of the fmadio_markets binary running on the public full day NASDAQ Totalview5 PCAP file (40GB worth)

fmadio_markets ITCH Decode Complete

Verification

The most important part is when the AI completes the feedback loop and verifies what it generates is correct (or not) using real verified datasources, fixing its own bugs. No one likes an engineer who writes code and never checks it actually works, machine or otherwise.

There are 2 verification passes with the code, first is using FMADIO Market Data Gap detector which writes the PCAP Timestamp and Sequence number for every single message in CSV format. This is a quick sanity check of the decoder, ensuring every message has been processed using a 3rd party tool.

For this we generate the sequence numbers in CSV + tell Claude to compare it against the Parquet file it has generated, the test script is linked below.

https://github.com/fmadio/fmadio_markets/blob/main/protocol/nasdaq_md_itch5/validate/validate_real.lua

It effectively ingests the CSV into DuckDB and performs a diff against the fmadio_markets binary vs what the CSV it has ingested.

Claude Testing itself against 3rd party data.

The resulting test run via a Makefile is shown below. Claude is cheeky enough to call out our “3rd party artifact” in the FMADIO MDGap logic, how dare it!
(yes, we have fixed the bug)

Verifying ITCH Data with FMADIO PCAP_MDGAP

Finally a full protocol decode test using Wireshark’s tshark and the market protocol decoders. This outputs a stream of JSON reference data per below:

 tshark  -r ny4-xnas-tvitch-20230822.pcap -nn  -V -X lua_script:/mnt/git/omi-wireshark/Nasdaq/Nasdaq_PsxEquities_TotalView_Itch_v5_0_2018_Dissector.lua  -T ek  > ny4-xnas-tvitch-20230822.pcap.tshark.json

Then getting Claude to compare every field against whats in our Parquet file. It first flattens the nested JSON so it can be ingested into DuckDB.

The AI then runs duckdb with a diff algorithm to detect differences between the data, pretty smart.

With the the test completing as shown below.

Summary

In Part 1 we have demonstrated how to generate a NASDAQ TotalView ITCH v5 protocol decoder, writing the dataset to an Arrow Parquet file format. The PARQ file can then be processed with ease using SQL to extract relevant information as needed.

To put this all into perspective, this was completed over an an afternoon per below. Consumed about 1H of AI time, and cost just under $100 to build a complete ITCH Market Data Decoder with full source code.

Compare that to how many days / weeks it would take a human engineer to build and its clear the direction this industry will go.

NASDAQ ITCH Full Decoder for $100

All the code generated is public on our GitHub project

https://github.com/fmadio/fmadio_markets/tree/main

And this exercise was itself a preview of something bigger. We're building on exactly this workflow to let customers generate decoders for their own proprietary protocols and run them on FMADIO hardware, straight out to Parquet. More on that in Part 2 — where we connect this to the FMADIO capture device for decoding live data in real time.‍ ‍

Contact us at support@fmad.io for a discussion on leveraging AI for Network Observability and Monitoring!

Aaron Foo
Next

Packet Broker Hardware Timestamps: Getting the Most from Your Network's Timing Data