__group__	ticket	summary	component	version	milestone	type	owner	status	created	_changetime	_description	_reporter
Active Tickets	35	Slow down query-rate	crawler			enhancement		new	2013-12-09T19:53:56+01:00	2013-12-09T19:53:56+01:00	Don't overload auth servers	wander
Active Tickets	39	Add support for wildcarded domains	crawler			defect		new	2014-08-31T00:18:12+02:00	2014-08-31T01:29:55+02:00	"Example:

{{{
plindex.com. 1401573062.87
Got IP for ns1.transip.nl.: 80.69.69.69
Got IP for ns1.transip.nl.: 2a01:7c8:b::53
Got IP for ns0.transip.net.: 80.69.67.67
Got IP for ns0.transip.net.: 2a01:7c8:a::53
Got IP for ns2.transip.eu.: 217.115.203.194
Got IP for ns2.transip.eu.: 2001:14a0:100:6::53
Connecting to database...
?Lucky hit: plindex.com.
?Lucky hit: plindex.com.
Assuming closest encloser plindex.com. -> EH7DC386J9T5AKS4VA1EEII8CCNBN67T
Assuming wildcard *.plindex.com. -> K06490OBB70PKKMD3N38JRJB1PISSH8R
paramset: ParamSet[NSEC3, Algorithm=1, Flags=None, Iterations=100, Salt=77a6ea5cde8c8255
paramid: 5602
Loading missing Gaps for paramid=5602 from database...
No ranges in database yet
Fresh run
?Lucky hit: vh6e2g04fh.plindex.com.
Traceback (most recent call last):
  File ""crawler.py"", line 1040, in <module>
    freshrun()
  File ""crawler.py"", line 614, in freshrun
    (serial, nseclist) = extract_nsec(response)
  File ""crawler.py"", line 742, in extract_nsec
    assert serial is not None, ""SOA not found""
AssertionError: SOA not found
}}}
"	wander
Active Tickets	5	Automatic crawling and job control	distserver			enhancement		new	2013-03-07T00:33:45+01:00	2013-03-07T00:33:45+01:00	"Automatic control script:
 * crawl missing zones (start from root)
 * re-crawl existing zones occasionally
 * create new dist projects for unknown hashes
"	wander
Active Tickets	9	Rebuild hashblobs from time to time when hashes have been found	distserver			enhancement		new	2013-03-07T11:53:13+01:00	2013-03-07T11:53:13+01:00	When hashblob is empty, stop project.	wander
Active Tickets	17	Brute-force closest enclosers and corresponding wildcard records	distserver			enhancement		new	2013-05-02T15:41:18+02:00	2013-05-02T15:41:18+02:00	"When attacking the NSEC3 hashes of a zone, make sure to try all possible closest enclosers and their corresponding wildcard records.

See http://tools.ietf.org/html/rfc5155#section-7.2 and Section 8.

Also: for short names up to 5 characters, attempt to brute-force all characters 0x00 to 0xFF? Might reveal whether anyone uses unusual characters. See also #16.

Depends on #5"	wander
Active Tickets	29	crawler: inconsistent data from different servers	crawler			enhancement		new	2013-05-20T03:38:08+02:00	2013-12-05T14:39:52+01:00	"Despite identical SOA serial the servers for com.br seem to deliver inconsistent NSEC records:

{{{
#?>!..#?=!..#?=!..#?.<!.#?.<!.#?.>!.#?.<!.#?>!..#?=!..?=!..#?
Inconsistent cut with serial 2013052006 FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing old NSEC FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM))
Right cut (NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))
!..#?>!..#?.<!.#?.<!.#?.|!.#?.<!.#?|!..#?|!..?|!..#?|!..#?
Inconsistent cut with serial 2013052006 FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Removing old NSEC FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Left cut (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI))
Removing (NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))
}}}

 * Notify operator of possible fault?
 * Add feature to prefer smaller (more specific) NSEC3 range in case of identical SOA serial?
"	wander
Active Tickets	37	Perform plausibility checks on results	distserver			enhancement	schwittmann	new	2014-01-28T14:23:19+01:00	2014-01-28T14:23:19+01:00	Job results should be checked e.g. if a hash(name) is really in the hashblob	schwittmann
Active Tickets	10	Change meaning of client 'accepted' flag to 'trusted'	nsec3breaker			enhancement		new	2013-04-02T11:24:55+02:00	2013-04-02T11:24:55+02:00	"Allow registration of clients by default, but set clients to trusted=false. If in doubt, untrusted results can be re-computed or run through anti-cheating.
"	wander
Active Tickets	16	Add dot to character set of candidate domain names	nsec3breaker			enhancement		new	2013-05-02T14:42:20+02:00	2013-05-02T14:42:20+02:00	"We use [a-z0-9-] as character set for candidate names. Some zones may use multi-label names, i.e. names with a dot.

Priority is nice-to-have, as the big TLD players won't use multi-label names in their zone.

Note 1: do '''not''' put the dot as 0x2E character into the binary representation, replace each '.' from presentation format into binary label length (see nsec3.name_to_bin(...)).

Note 2: other characters may be used in weird zones, too. Everything from 0x00 to 0xFF is allowed in DNS."	wander
Active Tickets	41	Rewrite Markov implementation with backtracking optimization	nsec3breaker			enhancement		new	2014-09-15T14:26:06+02:00	2014-09-15T14:26:06+02:00	"Current Markov implementation walks the prob tree each time for each candidate name (print_pwd function), and emits the word. A potentially major optimization is to walk the tree and emit words (call sha1 function) while keeping the backtracking state.

Also, consider a clean rewrite (we changed a hack, which mapped 0x00 to beginning of word and also mapped to a character in data structure, but this may have introduced a bug).
"	wander
Active Tickets	43	Allow setting the preferred job type (bf|dict|markov)	nsec3breaker			enhancement		new	2016-02-29T12:35:36+01:00	2016-02-29T12:35:36+01:00	Some platforms are better suitable for certain job types. Add an optional, manual client setting, which the distserver evaluates.	wander