Topic: Phage 0.0.1
For any of this to work, you need Freenet installed and pyFreenet as mentioned in the instructions. This is just a user script I wrote yesterday, don't take it too seriously - the point is more to show the concept. This script should be run about once a day to get the latest data. There is some code for managing and publishing samples on a periodic basis, I'll post that here in the near future once I feel better about how well it does its job.
For best results, run this script about once daily. This snippet here will check for new versions of the list of published samples and grab them for you. It will also fetch a list of all stored samples and show you only the ones which have been inserted into Freenet(and so are available to download).
I'm a bit behind schedule on releasing my application beta but I wanted to provide an update and give people a bit of a taste of what's in store.
Below is python script which listens to a USK. In every version of the USK, there are 4 key pieces of data:
samplelist - a dictionary of samples based on their sha256sum, a brief description of the file type, the size of the file and, if it exists, a chk of where to retrieve a sample of that file
insertreqs - this is a list that belongs to my identity of files I've either failed to retrieve or would like to be prioritized for insertion. This functionality hasn't been implemented yet
stats - a set of stats for the user, including the number and size of samples
recentinserts - a list of samples inserted since the last update
I'm starting with a collection of 38,000 ransomware samples.
You'll need pyFreenet 0.3.2 installed, which can be found in the Freenet indexes or at https://github.com/ArneBab/lib-pyFreenet-staging
The script can be downloaded from here: (see code listing below)
This is really just a hacky first attempt that can be improved significantly. I'm soon going to implement a python scraper for VirusTotal so we can have the detection ratio for each sample as well. I will eventually add a discussion section(potentially an FMS plugin), a place to share actual code listings instead of just samples and finally a marketplace.
After all of these concepts are done in PoC, I will re-implement in Java.
After installing pyFreenet, usage is:
python phage.py --grab-latest
python phage.py --list-all
python phage.py --list-inserted
Please let me know what successes and failures you have as well as suggestions. I know it needs a better interface.
One note about performance: The first run might take 10-15 minutes, as not many people are using this script(obviously). Freenet shares files more quickly as they become more popular, but since this is *not* popular, you might want to hit return on this script and go make yourself some lunch.
import fcp import pickle import sys #Including this class so we can read all samples class Sample: # Derive all of our needed information from a path def __init__(self, path): h = SHA256.new() f = open(path, 'rb') for l in f: h.update(l) self.path=path self.sha256sum=h.hexdigest() self.magictype=magic.from_file(path) self.size=os.lstat(path).st_size #self.detectratio=detectratio Add code for detection ratio later #Fetches a list of samples that have been recently inserted def recent_inserts(loc, node): recents = pickle.loads(node.get(uri=loc + "/recentinserts", followRedirect=True)) return recents def stats(loc, node): stats = pickle.loads(node.get(uri=loc + "/stats", followRedirect=True)) return stats def fetchallsamples(loc, node): samples = pickle.loads(n.get(uri=loc + "/samplelist", followRedirect=True)) return samples if __name__=="__main__": n = fcp.node.FCPNode() uri = "USK@~6vVyXEZElls5TBC5C~DXUB4eg2LIYX4ACgbpLMlxew,u-02Lf0DJbR2fjG2pjF2rXngLySjszgvv~L2349njD8,AQACAAE/phageproto/-1" print "Welcome to Phage 0.0.1!" if sys.argv == "--list-all": samples = fetchallsamples(uri, n) for v in samples.values(): print "SHA256SUM: " + v.sha256sum + ", " + v.magictype + ", " + str(v.size) if sys.argv == "--list-inserted": samples = fetchallsamples(uri, n) for k in samples.values(): try: print "SHA256SUM: " + k.sha256sum + ", " + k.magictype + ", " + k.chk + "," + str(k.size) except AttributeError: pass if sys.argv == "--grab-latest": newestinserts = recent_inserts(uri, n) for i in newestinserts: vxdata = n.get(i.chk, file=i.sha256sum)