__group__	ticket	summary	component	version	type	owner	status	created	_changetime	_description	_reporter
	35	Slow down query-rate	crawler		enhancement		new	2013-12-09T19:53:56+01:00	2013-12-09T19:53:56+01:00	Don't overload auth servers	wander
	39	Add support for wildcarded domains	crawler		defect		new	2014-08-31T00:18:12+02:00	2014-08-31T01:29:55+02:00	"Example:

{{{
plindex.com. 1401573062.87
Got IP for ns1.transip.nl.: 80.69.69.69
Got IP for ns1.transip.nl.: 2a01:7c8:b::53
Got IP for ns0.transip.net.: 80.69.67.67
Got IP for ns0.transip.net.: 2a01:7c8:a::53
Got IP for ns2.transip.eu.: 217.115.203.194
Got IP for ns2.transip.eu.: 2001:14a0:100:6::53
Connecting to database...
?Lucky hit: plindex.com.
?Lucky hit: plindex.com.
Assuming closest encloser plindex.com. -> EH7DC386J9T5AKS4VA1EEII8CCNBN67T
Assuming wildcard *.plindex.com. -> K06490OBB70PKKMD3N38JRJB1PISSH8R
paramset: ParamSet[NSEC3, Algorithm=1, Flags=None, Iterations=100, Salt=77a6ea5cde8c8255
paramid: 5602
Loading missing Gaps for paramid=5602 from database...
No ranges in database yet
Fresh run
?Lucky hit: vh6e2g04fh.plindex.com.
Traceback (most recent call last):
  File ""crawler.py"", line 1040, in <module>
    freshrun()
  File ""crawler.py"", line 614, in freshrun
    (serial, nseclist) = extract_nsec(response)
  File ""crawler.py"", line 742, in extract_nsec
    assert serial is not None, ""SOA not found""
AssertionError: SOA not found
}}}
"	wander
	5	Automatic crawling and job control	distserver		enhancement		new	2013-03-07T00:33:45+01:00	2013-03-07T00:33:45+01:00	"Automatic control script:
 * crawl missing zones (start from root)
 * re-crawl existing zones occasionally
 * create new dist projects for unknown hashes
"	wander
	9	Rebuild hashblobs from time to time when hashes have been found	distserver		enhancement		new	2013-03-07T11:53:13+01:00	2013-03-07T11:53:13+01:00	When hashblob is empty, stop project.	wander
	17	Brute-force closest enclosers and corresponding wildcard records	distserver		enhancement		new	2013-05-02T15:41:18+02:00	2013-05-02T15:41:18+02:00	"When attacking the NSEC3 hashes of a zone, make sure to try all possible closest enclosers and their corresponding wildcard records.

See http://tools.ietf.org/html/rfc5155#section-7.2 and Section 8.

Also: for short names up to 5 characters, attempt to brute-force all characters 0x00 to 0xFF? Might reveal whether anyone uses unusual characters. See also #16.

Depends on #5"	wander
	29	crawler: inconsistent data from different servers	crawler		enhancement		new	2013-05-20T03:38:08+02:00	2013-12-05T14:39:52+01:00	"Despite identical SOA serial the servers for com.br seem to deliver inconsistent NSEC records:

{{{
#?>!..#?=!..#?=!..#?.<!.#?.<!.#?.>!.#?.<!.#?>!..#?=!..?=!..#?
Inconsistent cut with serial 2013052006 FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing old NSEC FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM))
Right cut (NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))
!..#?>!..#?.<!.#?.<!.#?.|!.#?.<!.#?|!..#?|!..?|!..#?|!..#?
Inconsistent cut with serial 2013052006 FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Removing old NSEC FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Left cut (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI))
Removing (NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))
}}}

 * Notify operator of possible fault?
 * Add feature to prefer smaller (more specific) NSEC3 range in case of identical SOA serial?
"	wander
	37	Perform plausibility checks on results	distserver		enhancement	schwittmann	new	2014-01-28T14:23:19+01:00	2014-01-28T14:23:19+01:00	Job results should be checked e.g. if a hash(name) is really in the hashblob	schwittmann
	10	Change meaning of client 'accepted' flag to 'trusted'	nsec3breaker		enhancement		new	2013-04-02T11:24:55+02:00	2013-04-02T11:24:55+02:00	"Allow registration of clients by default, but set clients to trusted=false. If in doubt, untrusted results can be re-computed or run through anti-cheating.
"	wander
	16	Add dot to character set of candidate domain names	nsec3breaker		enhancement		new	2013-05-02T14:42:20+02:00	2013-05-02T14:42:20+02:00	"We use [a-z0-9-] as character set for candidate names. Some zones may use multi-label names, i.e. names with a dot.

Priority is nice-to-have, as the big TLD players won't use multi-label names in their zone.

Note 1: do '''not''' put the dot as 0x2E character into the binary representation, replace each '.' from presentation format into binary label length (see nsec3.name_to_bin(...)).

Note 2: other characters may be used in weird zones, too. Everything from 0x00 to 0xFF is allowed in DNS."	wander
	41	Rewrite Markov implementation with backtracking optimization	nsec3breaker		enhancement		new	2014-09-15T14:26:06+02:00	2014-09-15T14:26:06+02:00	"Current Markov implementation walks the prob tree each time for each candidate name (print_pwd function), and emits the word. A potentially major optimization is to walk the tree and emit words (call sha1 function) while keeping the backtracking state.

Also, consider a clean rewrite (we changed a hack, which mapped 0x00 to beginning of word and also mapped to a character in data structure, but this may have introduced a bug).
"	wander
	43	Allow setting the preferred job type (bf|dict|markov)	nsec3breaker		enhancement		new	2016-02-29T12:35:36+01:00	2016-02-29T12:35:36+01:00	Some platforms are better suitable for certain job types. Add an optional, manual client setting, which the distserver evaluates.	wander