Dd - Revision history

Leo: /* Copy while ignoring IO errors */

2022-03-20T23:55:20Z

Copy while ignoring IO errors

← Older revision		Revision as of 23:55, 20 March 2022
Line 19:		Line 19:

	===Copy while ignoring IO errors===		===Copy while ignoring IO errors===
	To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option.		To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option. See also [[Copy files from a failing disk\|my other page on copying files from a failing disk]].

	The {{code\|conv}} option specifies the type of conversion that dd needs to do to the input data. In this case, specifying {{code\|noerror}} 'converts' errors from the input medium into nulls in the output medium.		The {{code\|conv}} option specifies the type of conversion that dd needs to do to the input data. In this case, specifying {{code\|noerror}} 'converts' errors from the input medium into nulls in the output medium.

Leo at 05:05, 30 December 2021

2021-12-30T05:05:38Z

← Older revision		Revision as of 05:05, 30 December 2021
Line 3:		Line 3:
	{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.		{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.

	== Tasks ==		==Tasks==

	===Wiping a Disk===		===Wiping a Disk===
Line 82:		Line 82:
	}}		}}
	Re-read this block back afterwards and see if it's relocated.		Re-read this block back afterwards and see if it's relocated.

			=== Extracting resources from a binary ===
			See: [[Extract .exe Resources with dd]]

			=== Patching an executable with dd ===
			See: [[Patching a binary file with dd]]

	==See Also==		==See Also==

Leo at 05:02, 30 December 2021

2021-12-30T05:02:20Z

← Older revision		Revision as of 05:02, 30 December 2021
Line 3:		Line 3:
	{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.		{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.

	==~~Wiping a Disk~~==		== Tasks ==

	Since UEFI, GPT, and RAID information is typically saved to both the start and end of a disk, in order to clear the metadata, you'll have to write to both ends of the disk.		===Wiping a Disk===

			Since UEFI, GPT, and RAID information is typically saved to both the start and end of a disk, in order to clear the metadata, you'll have to write to both ends of the disk.

	To wipe the first and last 1MB of a disk:		To wipe the first and last 1MB of a disk:
Line 14:		Line 16:
	## Wipe last 1MB		## Wipe last 1MB
	# dd bs=512 if=/dev/zero of=/dev/sdb count=2048 seek=$((`blockdev --getsz /dev/sdb` - 2048))		# dd bs=512 if=/dev/zero of=/dev/sdb count=2048 seek=$((`blockdev --getsz /dev/sdb` - 2048))
	}}		}}There are other ways to wipe a disk on the [[Securely Erasing a Disk]] page.

	==~~Ignoring Errors~~==		===Copy while ignoring IO errors===
	To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option.		To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option.

Line 27:		Line 29:
	There are lots of other conversions that dd can handle such as case conversion, EBCDIC, byte swapping, and more. (see https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation)		There are lots of other conversions that dd can handle such as case conversion, EBCDIC, byte swapping, and more. (see https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation)

	==Manipulating by ~~Bytes~~==		===Manipulating data byte-by-byte===
	Use the {{code\|1=iflag=skip_bytes,count_bytes}} flags when working with data in bytes. The {{code\|bs}} block size value will be ignored when these flags are used.		Use the {{code\|1=iflag=skip_bytes,count_bytes}} flags when working with data in bytes. The {{code\|bs}} block size value will be ignored when these flags are used as the skip and offset values will be in bytes.
			{\| class="wikitable"
	~~Question: When using the~~ {~~{code\|count_bytes}} flag, is this equivalent to {{code~~\|1=~~bs=1}}? (It looks like it~~...)		!If you need...
			!Example command
	~~===Offset / Skipping Bytes===~~		\|-
	~~If you need to extract a chunk of data, say~~ everything past byte 199232000 from a file:		\|everything past byte 199232000 from a file:
			\|{{highlight\|lang=terminal\|code=
	{{highlight\|lang=terminal\|code=
	$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 {{!}} xxd		$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 {{!}} xxd
	}}		}}
			\|-
	~~To get~~ only 200 bytes of data ~~instead~~:		\|only 200 bytes of data starting at 199232000:
	{{highlight\|lang=terminal\|code=		\|{{highlight\|lang=terminal\|code=
	$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 count=200 {{!}} xxd		$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 count=200 {{!}} xxd
	}}		}}
			\|-
	Or 2 kilobytes (2048 bytes):		\|only 2 kilobytes (2048 bytes) at 199232000:
	{{highlight\|lang=terminal\|code=		\|{{highlight\|lang=terminal\|code=
	$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 count=2k {{!}} xxd		$ dd if=blah.dat iflag=skip_bytes,count_bytes skip=199232000 count=2k {{!}} xxd
	}}		}}
			\|}

	====Use-Case Example====		====Use-Case Example====
Line 64:		Line 66:
	This time, {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction and while the file is 'broken', the {{code\|-kb}} option kept the file.		This time, {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction and while the file is 'broken', the {{code\|-kb}} option kept the file.

	==== Write to a specific LBA ====		===Write to a specific LBA===
	~~If you don't have~~ [[hdparm]], you can use this method to write to a specific LBA. For example~~, if SMART is reporting an error on a specific LBA~~:		You may wish to write to a specific LBA to force a hard drive to reallocate a bad sector. You can either use [[Hdparm#Fix a damaged sector by forcing reallocation\|hdparm --write-sector ###]] or you can use this method using <code>dd</code> (though, hdparm is probably better suited for this task).

			If you know the bad LBA (such as from a SMART test, as shown below), you can write to the specific LBA by using its offset. For example:
	{{Highlight		{{Highlight
	\| code = SMART Self-test log structure revision number 1		\| code = SMART Self-test log structure revision number 1
Line 72:		Line 76:
	\| lang = text		\| lang = text
	}}		}}
	~~You~~ can write to ~~that specific~~ LBA ~~by using it as the offset and using~~ 512 block sizes~~. You might want to do~~ this ~~if (for example) you want to see if the disk can relocate the~~ bad block ~~elsewhere.~~		Knowing that each sector is 512 bytes and knowing the LBA of the first error, we can tell dd to write to the LBA location with 512 block sizes which should overwrite this bad block:
	{{Highlight		{{Highlight
	\| code = # dd if=/dev/zero of=/dev/sdb skip=4005685760 bs=512 conv=sync		\| code = # dd if=/dev/zero of=/dev/sdb skip=4005685760 bs=512 conv=sync

Leo: /* Use-Case Example */

2020-07-31T23:46:39Z

Use-Case Example

← Older revision		Revision as of 23:46, 31 July 2020
Line 1:		Line 1:
	{{DISPLAYTITLE:dd}}~~{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.~~		{{DISPLAYTITLE:dd}}

	== Wiping a Disk ==		{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.

			==Wiping a Disk==

	Since UEFI, GPT, and RAID information is typically saved to both the start and end of a disk, in order to clear the metadata, you'll have to write to both ends of the disk.		Since UEFI, GPT, and RAID information is typically saved to both the start and end of a disk, in order to clear the metadata, you'll have to write to both ends of the disk.
Line 14:		Line 16:
	}}		}}

	== Ignoring Errors ==		==Ignoring Errors==
	To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option.		To make dd ignore errors, such as when cloning a failing drive, add the {{code\|1=conv=noerror}} option.

Line 25:		Line 27:
	There are lots of other conversions that dd can handle such as case conversion, EBCDIC, byte swapping, and more. (see https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation)		There are lots of other conversions that dd can handle such as case conversion, EBCDIC, byte swapping, and more. (see https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation)

	== Manipulating by Bytes ==		==Manipulating by Bytes==
	Use the {{code\|1=iflag=skip_bytes,count_bytes}} flags when working with data in bytes. The {{code\|bs}} block size value will be ignored when these flags are used.		Use the {{code\|1=iflag=skip_bytes,count_bytes}} flags when working with data in bytes. The {{code\|bs}} block size value will be ignored when these flags are used.

	Question: When using the {{code\|count_bytes}} flag, is this equivalent to {{code\|1=bs=1}}? (It looks like it...)		Question: When using the {{code\|count_bytes}} flag, is this equivalent to {{code\|1=bs=1}}? (It looks like it...)

	=== Offset / Skipping Bytes ===		===Offset / Skipping Bytes===
	If you need to extract a chunk of data, say everything past byte 199232000 from a file:		If you need to extract a chunk of data, say everything past byte 199232000 from a file:

Line 47:		Line 49:
	}}		}}

	==== Use-Case Example ====		====Use-Case Example====
	I had a broken multi-archive .rar archive which was incomplete. {{code\|rar}} didn't like the file and refused to extract the file past this broken archive:		I had a broken multi-archive .rar archive which was incomplete. {{code\|rar}} didn't like the file and refused to extract the file past this broken archive:

Line 62:		Line 64:
	This time, {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction and while the file is 'broken', the {{code\|-kb}} option kept the file.		This time, {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction and while the file is 'broken', the {{code\|-kb}} option kept the file.

	== See Also ==		==== Write to a specific LBA ====
	* https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html		If you don't have [[hdparm]], you can use this method to write to a specific LBA. For example, if SMART is reporting an error on a specific LBA:
			{{Highlight
			\| code = SMART Self-test log structure revision number 1
			Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
			1 Extended offline Completed: read failure 40% 12786 4005685760
			\| lang = text
			}}
			You can write to that specific LBA by using it as the offset and using 512 block sizes. You might want to do this if (for example) you want to see if the disk can relocate the bad block elsewhere.
			{{Highlight
			\| code = # dd if=/dev/zero of=/dev/sdb skip=4005685760 bs=512 conv=sync
			\| lang = terminal
			}}
			Re-read this block back afterwards and see if it's relocated.

			==See Also==

			*https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html

	{{Navbox Linux}}[[Category:Linux]]		{{Navbox Linux}}
			[[Category:Linux]]
	[[Category:LinuxUtilities]]		[[Category:LinuxUtilities]]

Leo at 21:53, 14 June 2020

2020-06-14T21:53:01Z

← Older revision		Revision as of 21:53, 14 June 2020
Line 64:		Line 64:
	== See Also ==		== See Also ==
	* https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html		* https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html


	{{Navbox Linux}}[[Category:Linux]]		{{Navbox Linux}}[[Category:Linux]]
			[[Category:LinuxUtilities]]

Leo: Text replacement - "Category:Linux{{Navbox Linux}}" to "{{Navbox Linux}}Category:Linux"

2019-09-01T06:21:35Z

Text replacement - "Category:Linux{{Navbox Linux}}" to "{{Navbox Linux}}Category:Linux"

← Older revision		Revision as of 06:21, 1 September 2019
Line 66:		Line 66:


	[[Category:Linux]]~~{{Navbox Linux}}~~		{{Navbox Linux}}[[Category:Linux]]

Leo: Text replacement - "Category:Linux" to "Category:Linux{{Navbox Linux}}"

2019-09-01T05:08:42Z

Text replacement - "Category:Linux" to "Category:Linux{{Navbox Linux}}"

← Older revision		Revision as of 05:08, 1 September 2019
Line 66:		Line 66:


	[[Category:Linux]]		[[Category:Linux]]{{Navbox Linux}}

Leo: display title

2017-04-11T20:53:58Z

display title

← Older revision		Revision as of 20:53, 11 April 2017
Line 1:		Line 1:
	{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.		{{DISPLAYTITLE:dd}}{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.

	== Wiping a Disk ==		== Wiping a Disk ==

Leo: wiping start/end of a disk

2017-04-11T16:09:27Z

wiping start/end of a disk

← Older revision		Revision as of 16:09, 11 April 2017
Line 1:		Line 1:
	{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.		{{code\|dd}} is a utility that lets you copy data from one device/file to another while letting you change the block size and perform conversions if desired.

			== Wiping a Disk ==

			Since UEFI, GPT, and RAID information is typically saved to both the start and end of a disk, in order to clear the metadata, you'll have to write to both ends of the disk.

			To wipe the first and last 1MB of a disk:
			{{highlight\|lang=terminal\|code=
			## Wipe first 1MB
			# dd bs=512 if=/dev/zero of=/dev/sdb count=2048

			## Wipe last 1MB
			# dd bs=512 if=/dev/zero of=/dev/sdb count=2048 seek=$((`blockdev --getsz /dev/sdb` - 2048))
			}}

	== Ignoring Errors ==		== Ignoring Errors ==

Leo: /* Examples */

2017-04-10T17:31:56Z

Examples

← Older revision		Revision as of 17:31, 10 April 2017
Line 34:		Line 34:
	}}		}}

	==== ~~Examples~~ ====		==== Use-Case Example ====
	I had a broken multi-archive .rar archive which was incomplete ~~at the end~~. {{code\|rar}} didn't like the file and refused to extract the file past this broken archive. I appended the difference from the archive before it so that the file is 'complete' and then retried extracting the file again. {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction.		I had a broken multi-archive .rar archive which was incomplete. {{code\|rar}} didn't like the file and refused to extract the file past this broken archive:

			{{highlight\|lang=terminal\|code=
			$ rar e -kb something.iso.part001.rar something.iso
			...
			something.iso - checksum error
			Unexpected end of archive
			something.iso : packed data checksum error in volume something.iso.part009.rar
			}}

			I needed the file, even if it was corrupt in the middle. {{code\|rar}}'s repair tool didn't seem to help either. So, I appended the difference from the archive before it so that the file is 'complete' (ie. it has the same size as the other part archives) and then retried extracting the file again.

			This time, {{code\|rar}} complained {{code\|packed data checksum error in volume}} but continued on with the extraction and while the file is 'broken', the {{code\|-kb}} option kept the file.

	== See Also ==		== See Also ==