Round 2... 10 Trillion Digits of Pi

Same program, same computer, just a longer wait...

 

By Alexander J. Yee & Shigeru Kondo

 

(Last updated: December 28, 2013)

 

December 28, 2013: The record has been improved to 12 trillion digits.

 

For anyone who is interested, this was recently the topic of a Stack Overflow question that I answered.

Although not as exciting as the first time around, we finally managed to pull off a second run using same program and mostly the same hardware. This time, it took about a year due numerous hardware failures along the way...

 

3.
1415926535 8979323846 2643383279 5028841971 6939937510 : 50
5820974944 5923078164 0628620899 8628034825 3421170679 : 100
2597691971 6538537682 7963082950 0909387733 3987211875 : 4,999,999,999,950
6399906735 0873400641 7497120374 4023826421 9484283852 : 5,000,000,000,000
0874753286 1800417105 0417234177 3440660835 6303291247 : 5,000,000,000,050
0494788787 3350953767 0283816394 7603993291 8259328455 : 5,000,000,000,100
4974817456 4378784878 2784442724 8702748355 5412518205 : 6,000,000,000,000
3820441376 6006093732 4607915193 1761308978 1570965271 : 6,000,000,000,050
1248405812 4259631936 5227923837 0799098542 1979648580 : 7,000,000,000,000
1792125328 5414275137 5682812824 6229793359 0061969684 : 7,000,000,000,050
6636168271 2799313869 3626137973 6405361590 7572836055 : 8,000,000,000,000
6695601471 2802778702 1660409717 9930049024 4234355462 : 8,000,000,000,050
5399802500 0358265067 6788634714 4962622491 5222520345 : 9,000,000,000,000
7793366495 1565215134 1259428770 6232189125 1045517921 : 9,000,000,000,050
9544408882 6291921295 9268257225 1615742394 7483010753 : 9,999,999,999,950
9804871001 5982157822 2070871138 6966940952 1989228675 : 10,000,000,000,000
4392476662 7656619000 2124460557 5531593458 4820611421 : 10,000,000,000,050

 

Computation Statistics - All times are Japan Standard Time (JST).

 

Just like last time, the full computation statistics are listed below. Likewise, the final screenshot is shown. However, due to the numerous times that the computation had been interrupted, the screenshot only shows the final leg of the computation as opposed to start -> finish.

The main computation took 371 days on Shigeru Kondo's desktop. The computer was dedicated for this task.

 

Over the course of the computation, there were multiple hard drive failures. Each of which required us to roll the computation back to a previous checkpoint. The result was approximately 180 days of lost time.

 

Due to the size of this computation, a tremendous amount of memory was needed:

  - Roughly 44 TB* of disk was needed to perform the computation.

  - Another 7.6 TB of disk was needed to store the compressed output of decimal and hexadecimal digits.

 

If the digits were stored in an uncompressed ascii text file, the combined size of the decimal and hexadecimal digits would be 16.6 TB.

 

*All units used in this article are binary:

GB = 230 bytes

TB = 240 bytes

Pi - Computation
371 days
Start : 10:34 AM (JST) October 10, 2010
Finish:  3:14 PM (JST) October 16, 2011
Pi - Verification
1.86 days (Primary - Bellard's BBP)
4.94 days (Secondary - Plouffe's BBP)
Start :  See Detailed Timeline
Finish:  See Detailed Timeline

 

Computation Step Time
Series Summation: 157 days
Square Root: 5.57 days
Final Multiply: 3.57 days
Verify Multiply: 8.29 hours
Base Conversion: 18.0 days
Verify Base Conversion: 94.6 hours
Time Lost to HW Failures: 180 days
Total Computation Time: 191 days
Total Real Time: 371 days

 

Validation File: Validation - Pi - 10,000,000,000,050.txt


Note that the multicore efficiency % is inaccurate due to the numerous times that the computation has been interrupted.

For a detailed timeline of events related to this computation, see here.

 

 

Purpose

This computation was basically a followup to our computation to 5 trillion digits. So the motivation and purpose is the same.

 

To see how far can that limit of personal computing be pushed...

 

As we will show, reaching 10 trillion digits of Pi is much more difficult than 5 trillion digits using our current methods and 2010 computer hardware.

 

Hardware: Shigeru Kondo's Desktop

 

The machine we used is mostly the same as the previous computation. The only main difference is that there are more hard drives.

Processor
2 x Intel Xeon X5680 @ 3.33 GHz - (12 physical cores, 24 hyperthreaded)
Memory
96 GB DDR3 @ 1066 MHz - (12 x 8 GB - 6 channels)
Motherboard
Asus Z8PE-D12
Hard Drives
1 TB SATA II (Boot drive)
5 x 2 TB SATA II (Store Pi Output)
24 x 2 TB SATA II (Computation) - various models
Raid Controller
3 x LSI MegaRaid SAS 9260-8i
Operating System
Windows Server 2008 R2 Enterprise x64
Built By
Shigeru Kondo
Pictures
Click to enlarge.

 

Software: y-cruncher - A Multi-Threaded Pi Program/Benchmark

As with the hardware, the software side hasn't had any changes since our previous computation.

 

 

 

Software for Computation:

The program that was used for the main computation is y-cruncher v0.5.5.

Multiple builds were used between the different times that the computation was interrupted.

 

See main page: y-cruncher - A Multi-Threaded Pi Program

 

Version v0.5.5 is virtually identical to v0.5.4 as it consists only of a few bug fixes and adds a few improvements that are irrelevant to this computation.

 

Originally, our plan was to use v0.6.1 which is expected to be a significantly faster than v0.5.5. However, it was never completed in time for this computation. (and it still isn't complete...)

 

 

 

 

 

 

Software for Verification:

The program that was used for the verification is y-cruncher BBP v1.0.119.

 

See main page: y-cruncher BBP

 

This is the exact same binary that we used to verify our previous computation. There have been no changes at all.

 

 

This program implements the digit-extraction algorithm for Pi using the BBP formulas. It's sole purpose was to verify the main computation.

 

 

 

 

Both programs are written by me (Alexander J. Yee) and are available for download from their respective pages.

 

 

 

 

 


Methods/Formulas Used

Nothing has really changed from last time. See here for details.

 


10 Trillion Digits - Difficulties of this computation

More details coming later... after I finish my midterms...

 

 

Detailed Timeline (With Screenshots)

Date Event Screenshot
First Attempt
October 10, 2010

Start of Computation

View
October 16, 2010

Week 1: Series summation is past 6% complete.

View
October 23, 2010

Week 2: Series summation is past 13% complete.

View
October 30, 2010

Week 3: Series summation is past 17% complete.

View
November 6, 2010

Week 4: Series summation is past 22% complete.

View
November 14, 2010

Week 5: Series summation is past 28% complete.

View
November 20, 2010

Week 6: Series summation is still past 28% complete.

View
November 27, 2010

Week 7: Series summation is past 36% complete.

View
December 4, 2010

Week 8: Series summation is still past 36% complete.

View
December 9, 2010

Hard Drive Failure: Computation Failed

A backup was scheduled for when the 36% - 47% block was completed.

But a hard drive failed just before we got there...

View
     
Second Attempt
December 12, 2010

Start of Computation

View
December 18, 2010

Week 1: Series summation is past 6% complete.

View
December 25, 2010

Week 2: Series summation is past 13% complete.

View
January 1, 2011

Week 3: Series summation is past 17% complete.

View
January 8, 2011

Week 4: Series summation is past 22% complete.

View
January 15, 2011

Week 5: Series summation is past 28% complete.

View
January 22, 2011

Week 6: Series summation is still past 28% complete.

View
 

Checkpoint 4: Manual backup at 36%.

 
January 29, 2011

Week 7: Series summation is past 36% complete.

View
February 5, 2011

Week 8: Series summation is still past 36% complete.

View
February 6, 2011

Hard Drive Read Error: Roll back to Checkpoint 4.

View
February 11, 2011

Hard Drive Write Error: Roll back to Checkpoint 4.

All hard drives of the same model as the failed drive are replaced.

 
February 12, 2011

Week 9: Series summation is past 36% complete.

View
February 19, 2011

Week 10: Series summation is still past 36% complete.

View
February 26, 2011

Week 11: Series summation is still past 36% complete.

View
 

Checkpoint 3 Reached: Manual backup at 47%.

 
March 5, 2011

Week 12: Series summation is past 47% complete.

View
March 11, 2011

The Earthquake...

 

For a moment I pause to grasp what has happenned in Japan...

And the relief to have felt when I finally got that email from Shigeru Kondo

telling me that's he's okay.

 

Makes us wonder why we were still running this...

 
March 12, 2011

Week 13: Series summation is still past 47% complete.

 

If you're wondering how this was still running through all the power outages,

Japan is split into two power grids. Only the eastern grid (50 Hz) was affected

by the earthquake. The western grid (60 Hz) was not affected.

View
 

Hard Drive Read Error: Roll back to Checkpoint 3. (17 days lost)

 
March 19, 2011

Week 14: Series summation is past 47% complete.

View
March 27, 2011

Week 15: Series summation is still past 47% complete.

View
 

Hard Drive Write Error: Roll back to Checkpoint 3.

 
April 2, 2011

Week 16: Series summation is past 47% complete.

View
April 10, 2011

Hard Drive Failure: Roll back to Checkpoint 3. (again...)

 
April 16, 2011

Week 18: Series summation is past 47% complete.

View
April 23, 2011

Week 19: Series summation is still past 47% complete.

View
April 30, 2011

Week 20: Series summation is still past 47% complete.

View
May 2, 2011

Hard Drive Read Error: The program is able to recover and continue...

View
May 7, 2011

Hard Drive Read Error: Unrecoverable this time, roll back to Checkpoint 3.

All hard drives of the same model as the failed drive are replaced.

View
May 14, 2011

Week 22: Series summation is past 47% complete.

View
May 21, 2011

Week 23: Series summation is still past 47% complete.

View
 

Checkpoint 2 Reached: Manual backup at 60%.

 
May 28, 2011

Week 24: Series summation is past 60% complete.

View
June 4, 2011

Week 25: Series summation is still past 60% complete.

View
June 12, 2011

Week 26: Series summation is still past 60% complete.

View
June 18, 2011

Week 27: Series summation is still past 60% complete.

View
June 25, 2011

Checkpoint 1 Reached: Manual backup at 77%.

View
June 25, 2011

Week 28: Series summation is 77% complete.

July 2, 2011

Week 29: Series summation is past 77% complete.

View
July 10, 2011

Week 30: Series summation is still past 77% complete.

View
July 16, 2011

Week 31: Series summation is still past 77% complete.

View
July 23, 2011

Week 32: Series summation is still past 77% complete.

View
July 30, 2011

Week 33: Series summation is still past 77% complete.

View
August 4, 2011

Checkpoint 0 Reached: Manual backup at end of Series Summation.

 
August 6, 2011

Week 34: Final Division step of Series Summation.

View
August 13, 2011

Week 35: Final Multiplication step of Series Summation.

View
August 17, 2011

Series Completion Checkpoint: Manual backup at end of Series Computation

 
August 20, 2011

Week 36: Iterating Square Root.

View
August 26, 2011

Computation of Pi Complete: Output 8.3 trillion hexadecimal digits of Pi.

The digits match the BBP results!!!

View
View
August 27, 2011

Week 37: Base Conversion...

View
September 3, 2011

Week 38: Base Conversion...

View
September 10, 2011

Hard Drive Read Error: Roll back to end of series computation.

View
September 18, 2011

Week 40: Final Multiply...

View
 

Pi Computation Checkpoint: Manual backup at end of Pi computation.

 
September 25, 2011

Week 41: Base Conversion...

View
October 1, 2011

Week 42: Base Conversion...

View
October 8, 2011

Week 43: Base Conversion...

View
October 11, 2011

Base Conversion Complete: Output 10 trillion decimal digits of Pi.

The digits match our previous 5 trillion digit computation.

View
View
View
October 16, 2011

Verification Complete: End Computation of 10 trillion digits of Pi!!!

View

 

 


Questions or Comments

Contact us by e-mail.

 

You can also find me on XtremeSystems Forums under the username: poke349

Back To: