[R] handling breaks in a for loop

[Jeff Newmiller]

If your inout loop is ONLY storing data, then failures are either in the retrieval or storing actions. If you have sufficient storage space then the latter should be reliable. Network failures are however inevitable... you have no choice but to deal with them somehow... the internet is intrinsically unreliable. If missing one stock invalidates your analysis then you have set yourself up for failure and no amount of tricky coding will save you.

On February 6, 2022 11:41:54 AM PST, akshay kulkarni <akshay_e4 using hotmail.com> wrote:
>dear jeff,
>                restarting the storing of webscraped data should be easy, but what if it takes more than a few tries? I am using a very costly AWS EC2 instance and am hard pressed for funds. You should know that it is very difficult for me to afford even one break. I think the execution should take about two hours, but even one break could make it 5 - 6 hours (including fixing of the bug). What if I use save () in the body of the loop? It might make it run for about three hours, but it is better than 5 -6 hours right?
>
>THanking you,
>Yours sincerely,
>AKSHAY M KULKARNI
>________________________________
>From: Jeff Newmiller <jdnewmil using dcn.davis.ca.us>
>Sent: Monday, February 7, 2022 12:57 AM
>To: r-help using r-project.org <r-help using r-project.org>; Ebert,Timothy Aaron <tebert using ufl.edu>; akshay kulkarni <akshay_e4 using hotmail.com>; R help Mailing list <r-help using r-project.org>
>Subject: Re: [R] handling breaks in a for loop
>
>Perhaps a more conventional division of functional concerns should be applied? Separate collection of data from computation. Input functions should do input, analysis functions should do analysis, and output functions should do output... effectively lowering the cost of failure. If your first stage is simply storing the webscraped data, restarting should be easy.
>
>On February 6, 2022 11:03:19 AM PST, "Ebert,Timothy Aaron" <tebert using ufl.edu> wrote:
>>Is the bug in the data or in the code?
>>Can you identify the cause? Can you get a subset of the data to find the bug using fewer resources?
>>If you can identify the root cause, then you can set an error trap.
>>If the error is a memory problem, they you either need to save, reallocate memory, or get more memory.
>>
>>Tim
>>
>>-----Original Message-----
>>From: R-help <r-help-bounces using r-project.org> On Behalf Of akshay kulkarni
>>Sent: Sunday, February 6, 2022 1:56 PM
>>To: R help Mailing list <r-help using r-project.org>
>>Subject: [R] handling breaks in a for loop
>>
>>[External Email]
>>
>>dear members,
>>                         I have a very large for loop (basically a web scraping loop) with 500 iterations(over different stocks). I am concerned about the loop breaking midway.
>>
>>My question is: is there any method to store the succesful iterations and continue with the iteration (ofcourse, after fixing the bug) where the exception occured? I can insert a save () function in the body of the loop and save the succesful iterations but this would slow down the execution speed. Any other method? I want to avoid trycatch as much as possible because an NA in the place of a succesful iteration would cost me much.
>>
>>Thanking you,
>>Yours sincerely,
>>AKSHAY M KULKARNI
>>
>>        [[alternative HTML version deleted]]
>>
>>______________________________________________
>>R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Dhelp&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=jmZ4HPdnRH6ive_u-90RBNQQtFrnMey9AZv8MRZGQbc&e=
>>PLEASE do read the posting guide https://urldefense.proofpoint.com/v2/url?u=http-3A__www.R-2Dproject.org_posting-2Dguide.html&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=9PEhQh2kVeAsRzsn7AkP-g&m=F0cPNGUlUW-Cd9RJzYladHn1oypv0cPLA-d9dEO42hExw8rxis6aw3FPV6DtBQU2&s=Tpxb0boNRBRe0_7_FZteYKLLal9zfDXAXImIExuV35o&e=
>>and provide commented, minimal, self-contained, reproducible code.
>>
>>______________________________________________
>>R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>https://stat.ethz.ch/mailman/listinfo/r-help
>>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>and provide commented, minimal, self-contained, reproducible code.
>
>--
>Sent from my phone. Please excuse my brevity.

-- 
Sent from my phone. Please excuse my brevity.