Write-up: net100
Posted by Sander ‘dsc’ Ferdinand under HiTB 2017 CTF with tag(s) CTF

Introduction

Find the hidden message! download

TL;DR

Disclaimer

This challenge was made by our very own doskop.

Analysis

  • pcap file containing NNTP traffic (USENET)
  • Several files are being transferred and some of those are ‘split up’ in parts.
  • data itself is yEnc encoded. documentation

The objective for now is to find a way to get the files that are being transferred.

I struggled a bit with tshark to get those TCP streams out (tshark -r net100.pcap -T fields -e tcp.stream | sort -n | uniq), eventually settled with wireshark to export them. For this write-up, ill use tcpflow:

~$ tcpflow -r net100.pcap
~$ cat 192* > streams.raw

Now we have a single file containing the 4 TCP streams.

Decoding the data

Before writing our hacky Python script:

  • yEnc headers include: filename, filesize
  • yEnc headers mark the beginning of some data belonging to a file
  • yEnc headers may include an offset/limit range for partial transfers.

For our script we can iterate every line, upon embarking on lines that begin with ybegin record the proceeding data till yend - decode that data, store it in memory and after everything is done - write the files out.

For the decoding of yEnc data I used the Python library sabyenc. I used their example script on Github as a starting point.

#!/usr/bin/env python2
import re,sys,sabyenc

data = open(sys.argv[1], "r").readlines()
files = {}

for i in range(0, len(data)):
    line = data[i]

    if line.startswith("=ybegin"):  # beginning of a new yenc file
        has_offsets = data[i+1].startswith("=ypart")  # check if the second line contains begin/end
        headers = {  # nntp headers
            "begin": 0,
            "end": 0,
            "size": int(re.search('size=(.\d+?) ', line).group(1))
        }

        if not headers['size']: raise Exception("could not parse size=")
        if has_offsets:  # try to parse begin/end
            try:
                headers['begin'] = int(re.search('begin=(.\d+?) ', data[i+1]).group(1))
                headers['end'] = int(re.search('end=(.\d+?)\r\n', data[i+1]).group(1))
            except:
                print "warning: begin/end could not be correctly parsed"
        if not headers['begin']:  # sometimes begin turns into None, reset to 0
            headers['begin'] = 0

        # yencode-decoding using sabyenc - input data includes the 1/2 header lines and the trailing '=yend'
        decoded_data, output_filename, crc, crc_yenc, crc_correct = sabyenc.decode_usenet_chunks(
            data[i:], headers['size'])

        if not crc_correct: raise Exception("faulty checksum")
        print "output_filename:", output_filename
        print "size:", headers['size']
        print "decoded_data length:", len(decoded_data)

        if output_filename not in files:
            files[output_filename] = []
        files[output_filename].append({
            "begin": headers['begin'],
            "end": headers['end'],
            "data": decoded_data
        })

print "sorting chunks and writing files."

for output_filename, chunks in files.items():
    chunks = sorted(files[output_filename], key=lambda k: k['begin'])
    data = "".join([chunk["data"] for chunk in chunks])

    open("output/%s" % output_filename, "wb").write(data)
    print "written %s" % output_filename

I initially did not sort the data chunks which resulted in not having enough par2 recovery blocks.

par2

We end up with a bunch of .par2 files, which are recovery files.

22:57 $ ls -al ~/hitbctf/net100/output/
total 12752
drwxr-xr-x 2 dsc dsc    4096 Apr 16 22:33 .
drwxr-xr-x 4 dsc dsc    4096 Apr 16 22:33 ..
-rw-r--r-- 1 dsc dsc   40388 Apr 16 22:33 data.tar.bz2.par2
-rw-r--r-- 1 dsc dsc   45576 Apr 16 22:33 data.tar.bz2.vol0000+001.par2
-rw-r--r-- 1 dsc dsc   91048 Apr 16 22:33 data.tar.bz2.vol0001+002.par2
[...]

If we have enough recovery blocks we can use par2 to reconstruct the original data.tar.bz2 file. If anything went wrong in our Python script - we’ll see it in this stage.

~$ par2 r *

Repair is required.
1 file(s) are missing.
You have 1999 recovery blocks available.
Repair is possible.
1999 recovery blocks will be used to repair.
Wrote 10232329 bytes to disk

Repair complete.

After that we can extract files from the archive(s):

~$ bzip2 -d data.tar.bz2
~$ tar -xvf data.tar
~$ cd data

Which gives us 10.000 .dat files. lets grep for flags ¯\_(ツ)_/¯

~$ strings * | grep -oE "HITB{\w+}"

Well, that’s a lot of matches. Most of them don’t have numbers while flags usually do…

~$ strings * | grep -oE "HITB{\w+}" | grep -E "[0-9]"

1 result; the flag: HITB{1d4dd1ee96694cc44fe685d731f6bd2d}

additional solves

LuckY from eindbazen was able to whip out this one liner:

mkdir troep ; cd troep ; tcpflow -r ../net100.pcap ; uudeview *.bin ; par2repair

doskop made an all-solver: link