Forum: Author Hangout

A question about hard-drives and the data they hold.

Pixy 🚫

There are a lot of computer techs here and I, for one, am not. I do have a question though (well several actually, but moving on…)

My understanding, is that when you 'delete' something on your hard drive it's not actually deleted (unless you use specific software) all that happens is the (apologies, I don't know the correct terminology) 'index'/location for the data is removed so that it 'appears' to be deleted, but it actually isn't. Eventually it (the data) gets written over multiple times and I suppose its eventually lost. I say 'supposed' because I think that I read somewhere, sometime, that overwritten data can sometimes be recovered.

Now, my question is asked because of a plot point in a story I am playing with. The question being 'If you use standard disk image software (like, say, Samsung Magician), to copy the contents of one drive to another, does all that hidden data get copied as well, or just what's logged in the thing that tells the user what's on the drive?

For instance, my potential plot point is that a user 'deletes' all the stuff they don't want seen, then copies the drive and then replaces the original with the copy. If your average disk copy software copies over all the hidden data, then swapping out the drives is rather a pointless endeavour.

I know there are programs that can 'supposedly' secure delete a drive's contents, but how honest is that claim?

Also, is this the same for all OS's or is it different if you use Windows, Risc-Os, Linux, MacOS, etc etc.

I am aware that the only 'safe' way to delete data from a hard drive is to physically destroy the drive. However, the plot requirement is not the destruction of data, but the replacement of the drives with a sanitised version of the old ones, to give the illusion that nothing untoward is going on and everything is hunky-dory.

Replies: Dominions Son Lazeez Jiddan (Webmaster) Gauthier EricR Switch Blayde Pixy LupusDei

Dominions Son 🚫

@Pixy

I am not an expert in this sort of thing. The following is based on my understanding, but may be wrong.

A lot depends on what kind of drive you are talking about. It's going to be different for old magnetic platter drives vs newer solid state drives.

Recovery of genuinely deleted data is likely to be much harder with a solid state drive.

I say 'supposed' because I think that I read somewhere, sometime, that overwritten data can sometimes be recovered.

I've seen claims of being able to examine the platters and recover up to the last 10 values of any particular bit. However this involves disassembling the drive and examining individual platters with something on the order of an MRI or electron scanning microscope. A very expensive process.

This would not work with newer solid state drives.

I know there are programs that can 'supposedly' secure delete a drive's contents, but how honest is that claim?

It will depend on the specific program. The general idea behind such secure delete programs is to overwrite the data multiple times with all 1s or all 0s. If implemented properly, it will prevent recovery by anything short of the process I described above. That process would be extremely expensive and is not likely to be done outside of extreme cases.

If you are suspected of espionage, the government might go to such lengths, but that's about it.

And again, in a more recent setting, such recovery methods won't work on solid state drives.

Also, is this the same for all OS's or is it different if you use Windows, Risc-Os, Linux, MacOS, etc etc.

There might be minor differences, but the basic recovery/secure delete methods are based on the physical operation of the drives themselves.

Replies: REP

REP 🚫

@Dominions Son

Hello DS,

I an not an expert either, but I recall the following.

For solid state drives, the memory locations flip between 1 and 0. There is no image left behind on the digital circuitry.

That is not true of a magnet disk drive. Data is stored using a wright head on a spiral track as a series of 1s and 0s.. When writing the data, the 1s and 0s spill over the edges of the track. So the prior data can be recovered, if you have the proper equipment.

REP

Replies: Dominions Son

Dominions Son 🚫

@REP

That is not true of a magnet disk drive. Data is stored using a wright head on a spiral track as a series of 1s and 0s.. When writing the data, the 1s and 0s spill over the edges of the track. So the prior data can be recovered, if you have the proper equipment.

Yes, but from what I recall of that, the proper equipment is a high resolution MRI and/or an electron scanning microscope.

It can be done, but it's ridiculously expensive for the vast majority of use cases.

Lazeez Jiddan (Webmaster)

@Pixy

If your average disk copy software copies over all the hidden data, then swapping out the drives is rather a pointless endeavour

The average disk utility does not copy unused sectors.

There are several disk utilities that offer sector-by-sector disk copy as an option that the user must explicitly specify.

So your character can use one of these utilities while turning off the sector by section option and your plot would work.

Gauthier 🚫

@Pixy

It depends on:
* The OS.
* The File System
* The copy software
* The disk technology
* The sector size
* The block size
* The use of snapshots
* And don't forget to empty The recycle bin or bypass it...

On modern SSD when a file is deleted and is not part of an earlier snapshot, then the content may be (partially) erased by a TRIM if the blocks containing the file doesn't contain any other data.
On HDD the content is always left there accessible by raw disk access.

Normally on a Windows 10/11, a clone with magician will not copy unallocated sectors. So it may be clean.
But a program that erase free space would provide the exact same semi honest benefit... Accessing data written over with simple 0 is pure fiction unless you have a few millions dollars to trow away at a very long and likely unsuccessful recovery endeavor.

However, depending on the software used, there may be leftover data in a lot of places like:
* Volume Shadow Copies
* Temporary files
* Search index
* Automated screen captures (on Copilot+ PC)
* Cloud file history.
* Backup software (they often activate Volume Shadows Copies too).
* Recycle bin ;)

Which Magician will happily copy over and erase of free space will not touch.

Replies: madnige madnige

madnige 🚫

@Gauthier

For more completeness, I'll add another potential source of residual data: Self-Healing.

This is where the drive detects a failing sector, and permanently remaps a spare (reserved) sector in its place. The old sector data may or may not be copied to the new sector, but the old sector is left untouched thereafter, and is normally inaccessible - but could be accessed by recovery specialists or government operatives. It would not be touched by secure wipes or low-level copies - as far as the drive is concerned, it doesn't exist any more (unless the drive electronics are reprogrammed to resurrect it)

Another possible leak is 'recently used' lists, though these would not leak the data, rather just the possible existence of it.

madnige 🚫

@Gauthier

EricR 🚫

@Pixy

It totally depends on the software used for the copy.

As you know, erasure means removing the file system indexing and marking the sectors for recovery. Over time they will be overwritten. On a solid state system, once overwritten they are not recoverable. On a magnetic disk they may be recoverable depending on how many times they have been overwritten.

A standard file copy only copies files, not disk sectors. If it's not in the file index then it's not copied. A disk image copies sectors, whether they are organized and indexed as files or not. A disk image copies data that can be recovered.

There are variations in file systems to consider as well. NTFS, for example, maintains a bunch of metadata that can linger even after the file is deleted - journaling entries and other stuff. You may be able to detect the former presence of the file and determine information about it, but not the actual contents. It gets even more complicated on a Mac with APFS.

I think in your plot, it depends on how the drive is being used. If it's a system drive with a user file system then you have to image it to make sure the computer will run. But that will copy all the "deleted" data. If it's a data drive then you can simply copy the files to a clean drive without copying all the uncovered sectors and metadata.

Replies: solitude

solitude 🚫
Updated:

@EricR

On a solid state system, once overwritten they are not recoverable.

Not so simple. There's firmware on the drive that may be performing wear-levelling and other such tasks behind the back of the computer. The firmware maintains a hidden map that assigns virtual block numbers to actual blocks, and keeps a reserve of blocks in hand so it fan move things around and substitute good blocks for bad ones. This is why the OS is issuing 'trim' commands to the drive, so it knows when blocks are now free.

The overall result is that there may be hidden blocks that contain info that you thought had been deleted and overwritten. Not much, and special equipment has to be used... but if you had used that drive to store emails to/from Mr Epstein, you might be tempted to get out that hammer!

On the other hand, all these magic blocks are hidden away from the operating system by the firmware on the drive, so any software copying data from the drive won't see them.

By the way, some hard disks have a small amount of memory used to cache data being written out to speed up operations; this memory is persistent (it survives a power cycle) so that data is not lost if the system crashes at an awkward time. It is possible to think up pathological cases where a copy of info you thought had been deleted is actually still there. Again, it is the firmware on the drive that handles all this, and the computer can't access it.

(edited to correct a typo)

Replies: Michael Loucks

Michael Loucks 🚫

@solitude

The overall result is that there may be hidden blocks that contain info that you thought had been deleted and overwritten.

This is why you encrypt the drive completely (e.g., with FileVault on a Mac) before you write a single thing to the disk. Destroy the encryption keys, and that data becomes indistinguishable from random noise.

Note: If you do this on a Mac, do not allow Apple to store your keys or allow your drive to be unlocked with your iCloud password.

Switch Blayde 🚫

@Pixy

One solution is not to be too technical specific.

Let's say the plot requires the ability to discover data on a hard drive that's been deleted. You can always write that the FBI used a new technique that only a handful of people in the world knew about.

It requires suspension of disbelief, but many things in fiction require that — "Beam me up, Scotty."

Pixy 🚫

@Pixy

Thank you for all the replies. The protagonist or at least, the one doing the data recovery will be the Pentagon, so cost is not going to be too much of an issue, given the Pentagon's history of poor spending choices.

The owner of the drives is a media company/investigative journalists (who have ired the Pentagon). The main issue, or at least, the one I perceived would annoy all the techies the most, was going to be the transfer of the drives and the data held within. What I was looking at, was a way to transfer what was on the old drives to the 'new' drives so as to appear that the computers were still in possession of the 'original' drives, but lacking any content the media company didn't want the Pentagon to know they had (ie ongoing corruption/bribery investigations). Meanwhile, the original drives (and the precious data on them, including deleted data pertaining to previous investigations. In this case, it's not so much the old data which is the value (the investigations already having gone public) but the sources/names of the leakers) could be hidden elsewhere.

Given the time critical nature of things, they couldn't spend a great deal of time doing things (we are talking about 20 odd computers), so I have to be realistic as there is only going to one or two tech guys.

My first solution was to have them delete files pertaining to other investigations involving members of Congress/ other people in positions of power and then copy what was left on to the new drives. Hence why I wanted to know if disk copy programs copied everything on the platter, or just the data noted by the index file thingy that tells an OS where all the requisite data is.

I may need to look back into that part of the story and see if it can worded to involve a degree of 'handwavium', like SB said, that doesn't rage bait all the Tech-bros...

Replies: jimq2 storiesonline_23 Gauthier

jimq2 🚫

@Pixy

Depending on the copy program, most files will get a new origin date when created on the new drive. I had a SSD cloned onto a larger one, and all the files now have the same date.

storiesonline_23 🚫

@Pixy

I suggest that you decide on

- the degree of success that the Pentagon is to have
- how much effort the Pentagon is to expend
- how much care the media company has put into its information
security on an ongoing basis
- the computer infrastructure at the media company

Then you can call on all the potentially raging tech-bros here to
suggest what the one or two tech guys need to do so that the story can
unfold as it should. Your desired result may constrain the "upstream"
situation: sufficient ongoing cluelessness at the media company may
rule out inconvenience--let alone serious expense--for the Pentagon.

If you do not like what our tech-bros give you, you might consider
asking at https://www.schneier.com. The recurring "Friday Squid
Blogging" posts are fairly liberal in accepting comments.

There are companies specializing in data recovery. Whether one of
them would consult without big bucks, I cannot guess. At least there
used to be such companies. (Yikes, am I really that old? How did
that happen while I wasn't looking?)

To minimize strain on the reader's disbelief, you may want to give
some attention to the legal situation.

- Does the Pentagon look toward a prosecution, or does it just want
to know?
- How did the Pentagon get drives or drive images? Judicial warrant?
Black bag job? Something else?

I hope this helps.

(Yes, I *am* really good at thinking up work for other people. How did
you guess?)

Replies: EricR

EricR 🚫

@storiesonline_23

And… was any of the data stored in a cloud service. There are other attack vectors that wouldn't require the drives to be in the physical possession of the pentagon.

Replies: storiesonline_23

storiesonline_23 🚫
Updated:

@EricR

And… was any of the data stored in a cloud service.

Yup, that could well be "sufficient ongoing cluelessness". Can you say "National Security Letter"? Or a friendly insider--friendly, or coerced, or whatever--might be even more convenient.

An evil-maid attack could provide a more "active" scene for the story than paperwork would. OTOH, paperwork leaves room for a different kind of conflict later.

So many possibilities. And all this is without even knowing much of what the story is about. Am I turning into a raging tech-bro? Is that better or worse than being a rent-a-nerd and database-bithead?

Gauthier 🚫
Updated:

@Pixy

The owner of the drives is a media company/investigative journalists

Normally, those have top notch IT service which develop adequate countermeasures to protect source anonymity.

Moreover, there is extensive on the job training to address these issues. Each agency have their own way to ensure that no matter what, the source is protected.

Looking for "send a tip" across news organization will make it clear that preserving the source anonymity is rarely an afterthought.

Communication with the sources used to rely on the dark-net and stayed there. Today, signal is more used but it's a step backward.

However, with recent news, it looks like U.S. journalists have becomes complacent with their data and may overly rely on the judiciary protection they enjoyed for so long, before the new Nazi era.

so as to appear that the computers were still in possession of the 'original' drives

That is not possible at all. Forensic will detect the drive change trough multiple ways:

* Opening of the computer will leave physical trace.

* Power On time in the smart data will not match the computer age and the bytes written will be off.

* Analysis of the MFT or other data structure will reveal tempering.

* File location will be wrong as will be the lack of fragmentation.

* Reference to the old drive serial number will be littered everywhere: registry, eventlog, telemetry, crash data, not to mention possibly depending on the supplier, the invoice and bill of material...

* The clone tool will probably leave logs in multiple places.

...

LupusDei 🚫
Updated:

@Pixy

Random thoughts...

Yes, simple delete only mark the space the files occupy as available for reuse. That then happens on random basis, partial data may linger for prolonged time depending on write activity and relative disk space utilization.

Secure erase options:
a) overwrite all free space with zeroes once: fast, should be good enough, but probably recoverable in unlimited resources scenario depending on hardware in question;
b) overwrite free space with random stream data several times: taxing on time but supposedly military grade secure even on magnetic media.

In older windows with magnetic HDD simple defrag of a full enough drive may achieve almost b) of most space with high probability, but not so much on mostly empty disk.

Simple copy only transfer what's indexed. Creating disk image may possibly leak deleted data, if low level sector-by-sector copy is enabled.

Making a full disc copy should be faster than secure erase b). Depending on software used it may or not be possible to determine the new disk is indeed a copy.

It may be possible to determine the new disk is physically newer than the data it contains by simply examining the hardware and looking up its production date - to fool a serious investigation you may need to use hardware of similar physical age than the replaced.

Obscure leaks: it is sometimes possible to find garbage data within unused space of new and unrelated files - such as game save files - with fixed length structures left empty that do NOT overwrite older data in the operative memory or possibly the reused disk space the file is written over, but instead effectively capture that data and then perpetually transfer as part of the new file.

However, such garbage data fragments are accessible only by low level tools examining the file contents byte-by-byte (NOT the proper software using the file as intended), not easy to distinguish from whatever garbage the software itself created and I doubt there can be a structured methodology to discover and attribute such accidentally captured garbage data within existing files, but determined or lucky forensics may, if they must.

Reply to topic

Home | Forum | All Threads by Date

Forum Rules |

Forum: Author Hangout

A question about hard-drives and the data they hold.

WARNING! ADULT CONTENT...