Convert EnCase hash sets to md5sum
I managed to get a hold of a list of known-bad hashes to use in an experiment. The hashes, however, were in EnCase “.hash” format.
<div>
</div><div>I am mostly using the SleuthKit’s hfind to do some hash comparisons. My setup could already use the NSRL hash sets with no problem, and TSK is supposed to support EnCase hash sets. I was able to create an index for the EnCase hash sets, but when I attempted to query, I would get an error:</div><div>
</div><div>Command: hfind db.hash [hash value]
Error: “Cannot determine hash database type (hdb_setupindx: Unknown
Database Type in index header: encase)</div><div>
</div><div>No responses when asking about the error on the mailing list, so I looked for other ways to access the hashes.</div><div>
</div><div>Finally, I came across Jesse Kornblum’s EnCase hash file converter (encase2txt). The tool built fine in Linux (Ubuntu), and the Windows binary worked with no issue on Windows 7 (64bit).</div><div>
</div><div>Just point the tool at the EnCase hash database and it will output all the hashes in a format like md5sum. Pipe this plain text output to a file, and you have an md5sum hash file. From this I was able to build the index (hfind -i md5sum hashes.md5) and query the database with no problems.</div><div>
</div><div>Thanks Jesse!</div><div>
</div><div>Building and usage: http://jessekornblum.livejournal.com/166275.html</div>
[Live Stream] 5th International Conference on Digital Forensics & Cyber Crime ICDF2C 2013
The 5th International Conference on Digital Forensics and Cyber Crime (ICDF2C) will be held this week in Moscow (September 26-27, 2013). The program can be found here: http://d-forensics.org/2013/show/program-final
If you can’t make it to Moscow, you can access an online stream of the conference by following the instructions here (bottom of the page): http://d-forensics.org/2013/show/registration
I hope for a lot of people and interesting discussion!
Sending a manually encrypted email using a public key
| Some web-based email services don’t have an encryption client available, but if you still want to be able to encrypt an email using someone’s public key, you can do it in the following way. First, get GPG. <ul><li>There is GPG4Win: http://www.gpg4win.org/ (I’ve not used it, but I assume there is a command line client)</li><li>GPGTools for Mac: https://gpgtools.org/ (I’ve used, and liked very well)</li><li>For Linux there are few different options, so see this for more information.</li></ul> Once you have GPG installed, you should be able to run the command ‘gpg’ from a terminal. *note - On Windows or Mac the commands may be slightly different than what is shown here. For this tutorial, you will be encrypting an email using someone’s public key. I will not show you how to create your own keys in this single. Once you can run gpg, we need to try to find the public key of the person you will send the message to. If they have uploaded their key to a public key server you can query the server with <blockquote class="tr_bq">gpg –list-keys [email protected]</blockquote>You can then import the key you have found using –recv-keys with the key’s ID. For example, my key looks like: pub 2048R/606B15C4 2013-01-09 [expires: 2017-01-09]. The key’s ID is 606B15C4. <blockquote class="tr_bq">gpg –recv-keys 606B15C4 </blockquote>Once we have successfully imported the key, we can use it to encrypt messages that only the holder of the associated private key can decrypt. So, for example, if you use my public key to encrypt a file, then only my private key can be used to decrypt the file. Likewise, if I use my private key to encrypt a file, then only my public key can be used to decrypt the file. This is a good way to show that you are the originator of the information. OK - so let’s make some text to send in our email. I will create a text file as a simple file container. <blockquote class="tr_bq">echo “Question: Can security students encrypt their email?” >> encryption_test.txt echo “Hypothesis: Security students are too lazy to encrypt their email.” >> encryption_test.txt </blockquote>So now I have a plain text file called “encryption_test.txt” with two lines in it. (In Linux) I can read the contents of the file using ‘cat’, and pipe the output into gpg for encryption. <blockquote class="tr_bq">cat encryption_test.txt |
gpg –encrypt –armor -r 606B15C4</blockquote>The first command takes the contents of the file “encryption_test.txt” and send it to gpg, which encrypts the text with the public key 606B15C4. The output is as follows:-----BEGIN PGP MESSAGE-----If I copy from —BEGIN PGP MESSAGE — to —END PGP MESSAGE— and paste that into the body of my email, and send it to the person who owns the private key - they should be able to decrypt it. |
An Argument for Assumed Extra-territorial Consent During Cybercrime Investigations
As seen on DigitalFIRE.ucd.ie
During cybercrime investigations it’s common to find that a suspect has used technology in a country outside of the territorial jurisdiction of Law Enforcement investigating the case. The suspects themselves may also be located outside of the territory of the investigating group. A country may be able to claim jurisdiction over a suspect or device that is located outside of their territory [1], however, foreign Law Enforcement would not have jurisdiction within the territorial jurisdiction of another country unless explicitly granted. This means that if a suspect or digital device is located in another territory, the investigating country may need to request assistance from the country that has territorial jurisdiction. This request could be in the form of mutual legal assistance requests, international communication channels such as INTERPOL and United Nations networks, through a personal contact within the country of interest, etc.
<div class="separator" style="clear: both; text-align: center;">
</div>
It appears to be increasingly common that Law Enforcement will use personal contacts to quickly begin the investigation process in the country of interest and request data be preserved, while at the same time making an official request for cooperation through official channels. This is simply because official channels are currently far too slow to deal with many types of cybercrime that rely on preserving data before the records are overwritten or deleted; a problem that has been communicated by Law Enforcement for over a decade.
For similar reasons, Law Enforcement in many countries commonly access data stored on servers in countries outside of their jurisdiction. When and how they access this data is usually not well defined because law too, in most — if not all — countries, is failing to keep up with changes in cross-border digital crime. However, a recent work by the NATO Cooperative Cyber Defence Centre of Excellence — Tallinn Manual on the International Law Applicable to Cyber Warfare (Tallinn Manual) — attempted to explicitly state some of these issues and their practical implications, albeit in the context of Cyber Warfare.
In the Tallinn Manual the expert group considered issues of jurisdiction applied to cyber infrastructure. Of these considerations, they claim that “… States may exercise sovereign prerogatives over any cyber infrastructure located on their territory, as well as activities associated with that cyber infrastructure” [2] with some exceptions. Further, Rule 1 paragraph 8 stipulates that:
<blockquote class="tr_bq">A State may consent to cyber operations conducted from its territory or to remote cybercrime operations involving cyber infrastructure that is located on its territory.</blockquote>In this rule, the expert group gives the explicit example that a State may not have the technical ability to handle a situation within their territory, and thus may give permission for another State to conduct cyber activities within their jurisdiction.
Much of the discussion on sovereignty, jurisdiction and control stipulate the scope of control a State possesses; however, Rule 5 specifies the obligation of the State to other states. Specifically that “the principle of sovereign equality entails an obligation of all States to respect the territorial sovereignty of other States”. The expert group elaborates with Rule 5 paragraph 3 claiming that:
<blockquote class="tr_bq">The obligation to respect the sovereignty of another State… implies that a State may not `allow knowingly its territory to be used for acts contrary to the rights of other States’.</blockquote>Rule 5 paragraph 3 has interesting implications in cyber space. For example, the infrastructure of many different countries may be used in an attack against a single victim. Because of this rule, each country whose infrastructure was involved is obliged to not allow these attacks to continue once they are aware of such attacks. A State, however, is not necessarily obliged to actively look for attacks against other countries from its infrastructure.
In other words, if an attack is made from (or through) State A to State B, and State B makes State A aware of the attack, then State A is normally obliged to help in stopping — and presumably helping to investigate — the attack on State B, if possible.
The Tallinn Manual goes on with Rule 7 stating that an attack originating from a State is not proof of a State’s involvement, but “… is an indication that the Sate in question is associated with the operation”. However, instead of assuming that the State could be guilty, in this work we propose to assume the innocence of the state whose infrastructure is being used in an attack.
Let’s assume State B is affected by a cyber attack apparently originating from State A. State B then attempts to make State A aware of the attack. There is essentially one of three responses that State B will receive from State A: Response to collaborate, Response to not collaborate, or no response. In the case of no response if there is an assumption of innocence of State A, then State B may also assume that State A — being obliged to help — cannot stop the attacks because of lack of technical ability, resources, etc. In this way, consent to conduct remote cyber investigations on infrastructure within State A could potentially also be assumed.
In this way, when requests for assistance are made between States, if one State does not, or cannot, respond to the request, then cyber investigations can continue. Under this assumption, countries with intention to collaborate but limited investigation capacity, convoluted political and/or communication processes, or just no infrastructure will gain increased capacity to fight abuses of their infrastructure from countries that have more resources.
By assuming innocence of a state, at least four current problem areas can be improved. First, by assuming a State’s consent for remote investigation upon no reply to international assistance requests, this will lead to a reduction in delay during cross-border investigations for all involved countries despite weaknesses in bureaucratic official request channels. Second, such an assumption will force States to take a more active role in explicitly denying requests, if so desired, rather than just ignoring official requests, which is a waste of time and resources for everyone involved. Third, depending on the reason for the denial, such an explicit denial to investigate attacks against other countries would be slightly more conclusive proof of State A’s intention to attack, or allow attacks, on State B, and could potentially help where attack attribution is concerned. And finally, such an assumption may also hold where mutual legal assistance currently — and oftentimes — breaks down; when dual criminality does not exist between two countries [3].
Essentially, if an attack on Country B occurs from infrastructure in Country A, Country A will either want to help stop the attack or not. By assuming that Country A does want to help but is simply unable to, this forces Country A to be explicit about their stance on the situation while at the same time ensuring that international cybercrime investigations can be conducted in a timely manner.
James, J. I. (2013) “An Argument for Assumed Extra-territorial Consent During Cybercrime Investigations”. VFAC Review. Issue 25. [PDF]
Bibliography
<li>Malanczuk, P. (1997). Akehursts modern introduction to international law (7th ed.). Routledge.</li><li>Schmitt, M. N. (Ed.). (2013). Tallinn Manual on the International Law Applicable to Cyber Warfare. Cambridge University Press.</li><li>Harley, B. (2010). A Global Convention on Cybercrime?. Retrieved from http://www.stlr.org/2010/03/a-global-convention-on-cybercrime/</li>
Image courtesy of jscreationzs / FreeDigitalPhotos.net
CybercrimeTech Interviewed!: Baby Got Bactria
Since the single went up in the middle of travelling for conferences and Police training, I didn’t get a chance to thank the very interesting history blog Baby Got Bactria for interviewing Cybercrimetech.
<div>
</div><div>In the single, Briana K. asked us about issues facing Central Eurasia in Cyberspace. To which I went off on a rant. See the rant here!: http://bactrianbria.blogspot.kr/2013/06/central-eurasia-and-internet-lovehate.html
Thanks again Baby Got Bactria!
</div>
Using XARGS to speed up batch processing
| [Edit 24/7/2013] Be careful when using xargs to spawn multiple processes that write to the same file. I’ve been using it with md5sum and piping the output to a file. In data sets with a large number of files, several hundred hashes would be missing in the final file. A more reliable way is to use md5deep. In our last single we compared md5sum, md5deep and openssl md5 to determine which is fastest at calculating different hashes. For that experiment we were using 1 thread. Essentially, a file comes in, it gets hashes, the next file comes in. With 1 threat, we saw 1 out of the 4 processors being used. However, if we are processing a list of files, why not split the work into multiple jobs and use all of the processors. md5deep does this by default (4 threads), but splitting the jobs for any task is easily done with xargs. xargs is a program that allows you to control arguments coming in from a pipe. For example, if we type ‘find -type f |
md5sum’ then the list of found files would be hashed as one glob of data. To give each file name to md5sum, instead of all file names, we can use xargs to control how the output of find is piped to md5sum. For example, ‘find -type f | xargs -n1 md5sum’ will feed one line at a time to md5sum, allowing md5sum to find the file and hash it. You can also tell xargs how many threads to create using the -P switch. Since we have 4 processors, we will use 4 threads in this case. Consider in the last single, were the times for hashing all files in a directory were as follows: md5sum: real 3m46.117s md5deep: real 3m57.595s openssl: real 3m43.142s (example: ‘find -type f |
xargs -P4 -n1 md5sum’) Using xargs (and all of the processors), the real times can be reduced to: md5sum: real 1m57.196s md5deep: real 2m5.408s openssl: real 1m56.084s md5deep recursive, default threads (4), no xargs: real 2m0.521s Note: the user and system time is the same as before because the same amount of processor time is being devoted to the tasks, however, there are more processors so the real time is much less. Also, consider that all the files in the directory are different sizes, with the largest being 10GB. The majority of the time for each program was spent with 1 thread hashing the largest file, while the other processors had already finished hashing all the other files. |