Report on Resolving Some Known API Issues

This is a update to an earlier report which showed that many boost redeem
transactions that are referenced in the pow.co database are not known to the
Bitcoin miners.

Determination of the Overall Scope of the Problem

Aristotelis has written a script to check the database against the blockchain
and has determined that there are more than 1000 boosts of this type. This
problem started recently [find out how recent]

Investigation of Database

We looked into the database model and discovered that the database does not
store boost redeem transactions. Consequently, we could not examine any of these
transactions or attempt to broadcast them again.

In the process of the investigation, we discovered that the database does not
store enough informmation to reconstruct the boost proof. Right now someone who
wants to look at boosted content has to trust the API or go back to the
blockchain and check everything there. The Boost API ought to be able to serve
enough information to be able to check that all the boosts are
valid if the user wants to.

Is Insufficient Fees the Cause of the Bug?

Owen Kellog hypothesized that this problem has to do with transactions
containin insufficient miners fees that were dropped. This hypothesis can be
disproven because most of the boosts in the study were checked within 24 hours
of being created and no redeem tx was known for them at that time. Transactions
with insufficient fees are not dropped this quickly. It is true that we had
an earlier problem with txs produced with insufficient fees, but these have
all been mined into blocks because I bugged a miner about it. I have one
transaction that remaining that has yet to be mined in a block. It has
insufficient fees but it has stuck around in the mempool for months.

We do not know the original cause of the bug, but what we can say is that the
transactions that are referenced in the API database either were never broadcast
or were rejected as invalid immediately by the miners.

Analysis

The first thing that needs to be done is to prevent any new fake boosts from
being referenced in the database. In order to do this, the API should attempt
to broadcast the tx. If the tx can be broadcast or is already known to the
miners then it can be referenced in the database. Otherwise it should be
rejected.

It is possible that the miner who created these transactions
can find them and attempt to broadcast them. He has some time to do this
while the above repair is made to the API. Then all fake boosts redemptions
should be deleted from the database. These will then appear as unmined jobs to
be mined again.

The database model should be upgraded to include both the boost spending
transaction and the redemption transaction. The spending transactions should
be retrieved from the blockchain and provided to the database.

Finally, the API should be upgraded to enable users to request full boost proofs
for boosted content.

Boost Output Study

Abstract

We track a set of Boost transactions to analyze how many are dropped and under
what circumstances. We determine that many boost transactions are incorrectly
being marked as redeemed. This bug is extremely serious and the study suggests
that most of the boosts now appearing on pow.co are fake.

Method

Several boost transactions were created through the website pow.co. The txid was
collected off of the developer console. After one day, the boost transactions
were checked in the pow.co database and against the blockchain

Round 1

Data

The sample size is 21. The following txids are the boost transactions I generated.

outpoint	difficulty	profitability	spent
30a218b3acb1929fec9f23167903c4583fd93d22abe0e677c2b42d45c9817f79:0	0.0010000003662166597	791999.7099565117	true
ef8b76d65b57f23a868f0cda757601a86d40c2949f0eab8e02648ff8bc1f44dc:0	0.0010000003662166597	79244.5	true
735ca016acc893cfe90db58933b8452888e0a814c18dfcbdc59be0bfd7f84d51:0	0.0010000003662166597	705999.7414511329	true
283e8f1bf821958190f0076dc3553b88893b54ecfc40176944101773e1fc519f:0	0.06	86883.33333333333	true
e366486673c59327223b648bc18db96fa5e16f06a55a612ebe39f1960f894471:0	0.25	86892	true
9bbdf2159575fb52576d746622820e0b9be176b4242060df8e6518f4cec39f8b:0	0.002500010681360964	275998.8207827878	true
eddee9037110347047c05fe8ef6c8d38b221f4d32307a1dae0775ac7c6aa0840:0	0.0010000003662166597	739999.7289997711	false
32694d86d12f3caea01d757a3c86614ad225559ef04c6896e1cfef3d3b52b13e:0	0.012	62916.666666666664	true
accf7eafd240e0729b3c9999d3280651c4da15c3a680c04a142183651c20ba04:0	0.07000002136263718	79242.83295948729	false
488678f9b6f2403bfe78405d0c3a54033c73314d9380f1db3da1dc59c7284f13:0	0.9500028992230082	69018.73673609522	false
e48b0c47d729f1bd33548740a0a7233a9cea6f53d173c0330c0e088e4ed1c3b2:0	0.07000002136263718	65912.3710841535	true
9a965f097fb66b6f2d44c72806e33abe2bc438d40f56875e32664f131c32ab95:0	0.025	82960	true
ee6961e852e4c3e0ad8aea6bad7699ec39abff9c7b2ae4cc9d92b9ceb48ddb74:0	0.025	65920	true
8f690ab207df00cbf9af72e8b927869d475d5ac2e0b1b222c51e8a241dbe232e:0	1.5	72272	true
506f8d5e5955823dc8305732bcbebe7f464eb0b75fef157ed08ea6b6f20cce4f:0	0.0020000007324333193	1117999.5905699246	false
15ab60396bc3cf1f23967e8ac7f7641a2712e684b57dea3b01e8cf2af0755643:0	0.05550003810943335	72270.22064545663	true
174e07cc4e95354747eb9a7c1f347f2acd9eff7365670f6eec9c96591f88b3ab:0	1.7	60112.94117647059	true
3756e78341481a3320255dfa9f7e80280f6e3a74cb0c3a215bb39255b13c0a17:0	0.9900145023868512	60112.94117647059	true
9824cf95f0839ff4b8a7322138f2271f857a5cecbfee1040e1ce938048a97cd7:0	0.40000122072547717	75677.26904707408	true
084094641f80e71e6d00964790f8468d6c67c575c5c3a064f45598d3f98961c8:0	0.01	119900	false
fe46da3a259b81017ce429fa1a744e1661e4e3bc224a45fca2297ba27b091281:0	3.30000017	72272.11748901213	true

Results

One day later, five txs were found to be unspent. After provinding these txs to
the API, they remained unspent. The API says that they are spent when they are not spent.

round 2

Some upgrades were made to the API which may have resolved some issues, so a second study was made.

Data

Sample size is 5

outpoint	difficulty	profitability	spent
b844725cb234ed36d8df9baf454826e9159675ee91b1085b58f255f26a881360:0	0.4500010299863355	65913.18246738384	true
6e27c574483a129fb11b6ee221e5d0ca5942155ccff492f4213c7a41d7db7967:0	0.2100002883959099	60114.203158617536	true
cb27f1c204be1777ed4d5740fb72ce705ee5995062788f060a6f886b206823a2:0	1.00004577	1616.9259932972868	true
d5ab5c6dd06086c2277f9093b9d05ed488c948cc606948929fbbccdd9b6b98cd:0	1.00767267	612.3020087465507	true
e4960b51daf66c439eb28d42fdd1e66581dd8d45a74e735a859e0b414489031b:0	1.01547973	619.4116745195889	true

Results

All txs were mined. The sample size wasn't very big, so this may be coincedence.

Round 3

Sample size is 19

Data

outpoint	difficulty	profitability	spent
a956d11d92b923847061eecf3d58660a3b2c41c3b690c9bd12c32b04dc1f73ff:0	10	3539.7	false
018b31734f04fd2f8a3f937c071b7ea6e2e08853fc11184661f6e301d0f73d1f:0	1.30001388	9632.974072553749	false
c890f99123fd506934e8e500725527f6a136e7e4d0432dd034f34adf6036bf45:0	0.025	316240	false
62f83fdfbf44735695cff6f6846ebbe1970de50642db3a4d52b0ce0c188c47a8:0	0.025	138120	false
f62048cc9ff7dcd7e2544b8cff436a03f0a33d2c0dc61d3ef095232138d688d8:0	0.02	136250	false
8973cf7d2c87ef96a945ae687f759252ed92823a44652a35d4e58d123ae68fb0:0	0.025	110760	false
f8664d0988e0870928324ced1f5ea3b764ff6566013ec70267c61cd8e4b160a2:0	0.025	99200	true
5d6a2f236203beed9c162545340754c5e7395951098b78725a940d09e792c913:0	0.025	99200	false
0a1964e08342a248e09fedce3f68610e82b791881188555b91e2790a5bca3535:0	0.005000021362721928	516997.7911039902	false
55e4c9b1a251b46a56589a750cea0799adc44ddf09ffc4f5b720d9ee158937c7:0	0.025	104800	false
2a84f25f34b3b8220cd55ee65f585f26af17745eaf46004faf6b4545eb1e048f:0	0.04	108050	false
fc9aa0a774c8140b5cc4b4f3e1013973814ac754438700f44aafc28086b45996:0	0.03	103400	false
11298a827d7b06fb13a4feb932230f8dac8786da507810ab6b8e941e03d9f1f7:0	0.025	104800	false
4b7f07cba0b5684da075e778bf7623b6a9383aaf1a88d89068122ae61d13365b:0	0.025	110760	false
aa354f1e15970e14bb8678d32f0a0941bbe51d5fd44b68764ccc8ab977107da3:0	0.025	110760	true
9df1fbe58ccee6b5c9e81d8be4c746ca58dbd97853c06d84f7c7124454a851c2:0	0.002500010681360964	499997.8637369345	false
06665da7c65dd48fa1178398db085f33e90f02ffdd10893a7b465897d3195708:0	0.002500010681360964	499997.8637369345	false
3bd588090ef109c715b8873bbcf049c88e43050045c5c6711fd059d7972cc7c9:0	0.002500010681360964	447198.0893263142	false
6ee7eee9d0febb115899b2db8ad481f5f97cef960c6ad07b41ecb1e6e60dfd3c:0	0.025	88800	false

Results

Many transactions were not mined. The first two of these were not listed in the
database as having been spent but the rest were. Apparently these two were
overlooked due to having a difficulty higher than one and were not worked on.
The rest were incorrectly marked as spent, as before.

Analysis

From our analysis, a picture of a serious bug emerges. We now have a sample of
20 boost outputs that have been accepted by the API but not mined. The reason
they have not been mined is that they have been incorrectly marked as spent.

We investigated these transactions further to look for more details. When
returned by the API, each spent boost output comes with a spending txid and
input index. We show all 20 txs with the given spending txid. We investigated
whether these transactions are known to whatsonchain.com.

outpoint	given spending inpoint	known to whatsonchain.com
eddee9037110347047c05fe8ef6c8d38b221f4d32307a1dae0775ac7c6aa0840:0	1bb7dfd62cc615c5ec8354bd2dc30b1bc9fd6804eb1a9e31f2462e7f8409bd6f:0	false
accf7eafd240e0729b3c9999d3280651c4da15c3a680c04a142183651c20ba04:0	92158b48423b7129ea5e3f2020823344beaefd0636d2ecaab7d2a8b9ecf91c25:0	false
488678f9b6f2403bfe78405d0c3a54033c73314d9380f1db3da1dc59c7284f13:0	406fc5242032ec3f2c9b0ed5e6e3bb2622315d1a8b9bc2606d0f92354ed7fbeb:0	false
506f8d5e5955823dc8305732bcbebe7f464eb0b75fef157ed08ea6b6f20cce4f:0	40db57b23346c7be201049165cbff6c7865f62fc3ba81417521186e15285a33a:0	false
084094641f80e71e6d00964790f8468d6c67c575c5c3a064f45598d3f98961c8:0	f0343a9c311cc57eea6fccb16c8a0dfcf9ac7fdb387022ea087dffa8428767d7:0	false
c890f99123fd506934e8e500725527f6a136e7e4d0432dd034f34adf6036bf45:0	36025bf939deb07321d5852ee2cef5329a468f10407e5718da745d6fab87e13b:0	false
62f83fdfbf44735695cff6f6846ebbe1970de50642db3a4d52b0ce0c188c47a8:0	6b7dae96a79be6ce4ec3960a9235e5963c990c1467987932484698a96a9ebada:0	false
f62048cc9ff7dcd7e2544b8cff436a03f0a33d2c0dc61d3ef095232138d688d8:0	65f4112500d79604a2ef8bc2489e3376d0a64b1bdfb92d51e28d4c558b183857:0	false
8973cf7d2c87ef96a945ae687f759252ed92823a44652a35d4e58d123ae68fb0:0	7b1ba2cc0ec27c8e5f106fa7853f94659cb00d8cdbc1e3be213dd36c6cbb8530:0	false
5d6a2f236203beed9c162545340754c5e7395951098b78725a940d09e792c913:0	bc389aaee004f3bb0f9fd469db94db4db139457c563bb372cb1034b5ffd7c58e:0	false
0a1964e08342a248e09fedce3f68610e82b791881188555b91e2790a5bca3535:0	db6a07ad75a75001c0e2d14a86e27b048e75e4e9e96ca07476aa8b27fd3256c7:0	false
55e4c9b1a251b46a56589a750cea0799adc44ddf09ffc4f5b720d9ee158937c7:0	e6f9918982b4966ff89d451195836028b8c65830e12c90c303c1df114b3fed45:0	false
2a84f25f34b3b8220cd55ee65f585f26af17745eaf46004faf6b4545eb1e048f:0	1ebf240edc94ea40cfe530640ba87e29327f94e9450da5a2967d0f8d7bf5640d:0	false
fc9aa0a774c8140b5cc4b4f3e1013973814ac754438700f44aafc28086b45996:0	9be6fdd556ce0ff2784c1d83fa6b028c158862180021dc3e917d288bd383fc0c:0	false
11298a827d7b06fb13a4feb932230f8dac8786da507810ab6b8e941e03d9f1f7:0	00f2d0b2834ef019507a3b2d70e6678add3581472ffe8ee82b21b84946b0694a:0	false
4b7f07cba0b5684da075e778bf7623b6a9383aaf1a88d89068122ae61d13365b:0	c4d37509edd88b101faecd559b69194a39c1ffeb4409f4cb5318ec060d783999:0	false
9df1fbe58ccee6b5c9e81d8be4c746ca58dbd97853c06d84f7c7124454a851c2:0	ce5ff1825b27950b340b972f4328744bfd20977e7b1327b5dd7f6c9c0e4c4a5f:0	false
06665da7c65dd48fa1178398db085f33e90f02ffdd10893a7b465897d3195708:0	476d8a6c771495bd4c2aaf60a5862605890e1ef896d8c155484c0c49b6370cf3:0	false
3bd588090ef109c715b8873bbcf049c88e43050045c5c6711fd059d7972cc7c9:0	b5bb7818a2c7d765d57e356e69ede504fd3f079d4031ae9595c7412451495d81:0	false
6ee7eee9d0febb115899b2db8ad481f5f97cef960c6ad07b41ecb1e6e60dfd3c:0	6ee7eee9d0febb115899b2db8ad481f5f97cef960c6ad07b41ecb1e6e60dfd3c:0	false

Conclusion

It appears that transactions are being generated to redeem these boosts but
they do not end up on the blockchain.

Best case scenario is that these txs exist somewhere but were not broadcast.
Then the solution is to broadcast them. Worst case scenario is that the txs were
invalid. Then some bug needs to be identified and the database needs to be
scanned to delete all the fake txs.