Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nightmare: Segfault from freeing already freed memory when GPU=0 and iters>=2 #85

Open
imaami opened this issue Jul 15, 2017 · 0 comments

Comments

@imaami
Copy link

imaami commented Jul 15, 2017

(Notice: you'll probably want to patch this other bug first because you may not even get past network config parsing otherwise.)

When compiled with -DGPU=0, optimize_picture()in examples/nightmare.c sets net->input = im.data; on line 65 and then does free_image(im); on line 102 without ever clearing net->input. If iters==1 nothing bad happens, but if there are multiple iterations then resize_network() will do free(net->input); on that same already freed pointer during the next iteration.

The fix is to add net->input = NULL; after backward_network(*net); in optimize_picture().

Shell output:

$ ./darknet nightmare cfg/jnet-conv.cfg jnet-conv.weights data/scream.jpg 2 -iters 2
policy: Using default 'constant'
max_batches: Using default '0'
layer     filters    size              input                output
    0 conv     32  3 x 3 / 1    10 x  10 x   3   ->    10 x  10 x  32
    1 conv     32  3 x 3 / 1    10 x  10 x  32   ->    10 x  10 x  32
    2 max          2 x 2 / 2    10 x  10 x  32   ->     5 x   5 x  32
    3 conv     64  3 x 3 / 1     5 x   5 x  32   ->     5 x   5 x  64
    4 conv     64  3 x 3 / 1     5 x   5 x  64   ->     5 x   5 x  64
    5 max          2 x 2 / 2     5 x   5 x  64   ->     3 x   3 x  64
    6 conv    128  3 x 3 / 1     3 x   3 x  64   ->     3 x   3 x 128
    7 conv    128  3 x 3 / 1     3 x   3 x 128   ->     3 x   3 x 128
    8 max          2 x 2 / 2     3 x   3 x 128   ->     2 x   2 x 128
    9 conv    256  3 x 3 / 1     2 x   2 x 128   ->     2 x   2 x 256
   10 conv    256  3 x 3 / 1     2 x   2 x 256   ->     2 x   2 x 256
   11 max          2 x 2 / 2     2 x   2 x 256   ->     1 x   1 x 256
   12 conv    512  3 x 3 / 1     1 x   1 x 256   ->     1 x   1 x 512
   13 conv    512  3 x 3 / 1     1 x   1 x 512   ->     1 x   1 x 512
   14 max          2 x 2 / 2     1 x   1 x 512   ->     1 x   1 x 512
   15 conv   1024  3 x 3 / 1     1 x   1 x 512   ->     1 x   1 x1024
   16 conv   1024  3 x 3 / 1     1 x   1 x1024   ->     1 x   1 x1024
   17 max          2 x 2 / 2     1 x   1 x1024   ->     1 x   1 x1024
Loading weights from jnet-conv.weights...Done!
Iteration: 0, 1, Segmentation fault (core dumped)

GDB backtrace:

#0  __GI___libc_free (mem=0x7f01f4d70010) at malloc.c:2966
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = 0x0
#1  0x00005560520010a9 in resize_network (net=net@entry=0x7ffd0bff3c70, w=<optimized out>, w@entry=148, h=<optimized out>, h@entry=189) at ./src/network.c:374
        i = 3
        inputs = <optimized out>
        workspace_size = 32223744
        out = {type = MAXPOOL, activation = LOGISTIC, cost_type = SSE, forward = 0x55605201591e <forward_maxpool_layer>, backward = 0x556052015ace <backward_maxpool_layer>, update = 0x0, 
          forward_gpu = 0x0, backward_gpu = 0x0, update_gpu = 0x0, batch_normalize = 0, shortcut = 0, batch = 1, forced = 0, flipped = 0, inputs = 895104, outputs = 224960, nweights = 0, 
          nbiases = 0, extra = 0, truths = 0, h = 189, w = 148, c = 32, out_h = 95, out_w = 74, out_c = 32, n = 0, max_boxes = 0, groups = 0, size = 2, side = 0, stride = 2, reverse = 0, 
          flatten = 0, spatial = 0, pad = 0, sqrt = 0, flip = 0, index = 0, binary = 0, xnor = 0, steps = 0, hidden = 0, truth = 0, smooth = 0, dot = 0, angle = 0, jitter = 0, 
          saturation = 0, exposure = 0, shift = 0, ratio = 0, learning_rate_scale = 1, softmax = 0, classes = 0, coords = 0, background = 0, rescore = 0, objectness = 0, does_cost = 0, 
          joint = 0, noadjust = 0, reorg = 0, log = 0, tanh = 0, alpha = 0, beta = 0, kappa = 0, coord_scale = 0, object_scale = 0, noobject_scale = 0, mask_scale = 0, class_scale = 0, 
          bias_match = 0, random = 0, thresh = 0, classfix = 0, absolute = 0, onlyforward = 0, stopbackward = 0, dontload = 0, dontloadscales = 0, temperature = 0, probability = 0, 
          scale = 0, cweights = 0x0, indexes = 0x7f01efb9b010, input_layers = 0x0, input_sizes = 0x0, map = 0x0, rand = 0x0, cost = 0x0, state = 0x0, prev_state = 0x0, forgot_state = 0x0, 
          forgot_delta = 0x0, state_delta = 0x0, combine_cpu = 0x0, combine_delta_cpu = 0x0, concat = 0x0, concat_delta = 0x0, binary_weights = 0x0, biases = 0x0, bias_updates = 0x0, 
          scales = 0x0, scale_updates = 0x0, weights = 0x0, weight_updates = 0x0, delta = 0x7f01ef1f9010, output = 0x7f01ef6ca010, squared = 0x0, norms = 0x0, spatial_mean = 0x0, 
          mean = 0x0, variance = 0x0, mean_delta = 0x0, variance_delta = 0x0, rolling_mean = 0x0, rolling_variance = 0x0, x = 0x0, x_norm = 0x0, m = 0x0, v = 0x0, bias_m = 0x0, 
          bias_v = 0x0, scale_m = 0x0, scale_v = 0x0, z_cpu = 0x0, r_cpu = 0x0, h_cpu = 0x0, prev_state_cpu = 0x0, temp_cpu = 0x0, temp2_cpu = 0x0, temp3_cpu = 0x0, dh_cpu = 0x0, 
          hh_cpu = 0x0, prev_cell_cpu = 0x0, cell_cpu = 0x0, f_cpu = 0x0, i_cpu = 0x0, g_cpu = 0x0, o_cpu = 0x0, c_cpu = 0x0, dc_cpu = 0x0, binary_input = 0x0, input_layer = 0x0, 
          self_layer = 0x0, output_layer = 0x0, reset_layer = 0x0, update_layer = 0x0, state_layer = 0x0, input_gate_layer = 0x0, state_gate_layer = 0x0, input_save_layer = 0x0, 
          state_save_layer = 0x0, input_state_layer = 0x0, state_state_layer = 0x0, input_z_layer = 0x0, state_z_layer = 0x0, input_r_layer = 0x0, state_r_layer = 0x0, input_h_layer = 0x0, 
          state_h_layer = 0x0, wz = 0x0, uz = 0x0, wr = 0x0, ur = 0x0, wh = 0x0, uh = 0x0, uo = 0x0, wo = 0x0, uf = 0x0, wf = 0x0, ui = 0x0, wi = 0x0, ug = 0x0, wg = 0x0, 
          softmax_tree = 0x0, workspace_size = 0}
#2  0x0000556051fde007 in optimize_picture (net=net@entry=0x7ffd0bff3c70, orig=..., max_layer=max_layer@entry=2, scale=0.421875, rate=rate@entry=0.0399999991, thresh=thresh@entry=1, 
    norm=norm@entry=1) at ./examples/nightmare.c:42
        dx = 1
        dy = 5
        flip = 0
        crop = {w = 352, h = 448, c = 3, data = 0x556054142a40}
        im = {w = 148, h = 189, c = 3, data = 0x556054310a50}
        last = {type = MAXPOOL, activation = LOGISTIC, cost_type = SSE, forward = 0x55605201591e <forward_maxpool_layer>, backward = 0x556052015ace <backward_maxpool_layer>, update = 0x0, 
          forward_gpu = 0x0, backward_gpu = 0x0, update_gpu = 0x0, batch_normalize = 0, shortcut = 0, batch = 1, forced = 0, flipped = 0, inputs = 5046272, outputs = 1261568, nweights = 0, 
          nbiases = 0, extra = 0, truths = 0, h = 448, w = 352, c = 32, out_h = 224, out_w = 176, out_c = 32, n = 0, max_boxes = 0, groups = 0, size = 2, side = 0, stride = 2, reverse = 0, 
          flatten = 0, spatial = 0, pad = 0, sqrt = 0, flip = 0, index = 0, binary = 0, xnor = 0, steps = 0, hidden = 0, truth = 0, smooth = 0, dot = 0, angle = 0, jitter = 0, 
          saturation = 0, exposure = 0, shift = 0, ratio = 0, learning_rate_scale = 1, softmax = 0, classes = 0, coords = 0, background = 0, rescore = 0, objectness = 0, does_cost = 0, 
          joint = 0, noadjust = 0, reorg = 0, log = 0, tanh = 0, alpha = 0, beta = 0, kappa = 0, coord_scale = 0, object_scale = 0, noobject_scale = 0, mask_scale = 0, class_scale = 0, 
          bias_match = 0, random = 0, thresh = 0, classfix = 0, absolute = 0, onlyforward = 0, stopbackward = 0, dontload = 0, dontloadscales = 0, temperature = 0, probability = 0, 
          scale = 0, cweights = 0x0, indexes = 0x7f01efb9b010, input_layers = 0x0, input_sizes = 0x0, map = 0x0, rand = 0x0, cost = 0x0, state = 0x0, prev_state = 0x0, forgot_state = 0x0, 
          forgot_delta = 0x0, state_delta = 0x0, combine_cpu = 0x0, combine_delta_cpu = 0x0, concat = 0x0, concat_delta = 0x0, binary_weights = 0x0, biases = 0x0, bias_updates = 0x0, 
          scales = 0x0, scale_updates = 0x0, weights = 0x0, weight_updates = 0x0, delta = 0x7f01ef1f9010, output = 0x7f01ef6ca010, squared = 0x0, norms = 0x0, spatial_mean = 0x0, 
          mean = 0x0, variance = 0x0, mean_delta = 0x0, variance_delta = 0x0, rolling_mean = 0x0, rolling_variance = 0x0, x = 0x0, x_norm = 0x0, m = 0x0, v = 0x0, bias_m = 0x0, 
          bias_v = 0x0, scale_m = 0x0, scale_v = 0x0, z_cpu = 0x0, r_cpu = 0x0, h_cpu = 0x0, prev_state_cpu = 0x0, temp_cpu = 0x0, temp2_cpu = 0x0, temp3_cpu = 0x0, dh_cpu = 0x0, 
          hh_cpu = 0x0, prev_cell_cpu = 0x0, cell_cpu = 0x0, f_cpu = 0x0, i_cpu = 0x0, g_cpu = 0x0, o_cpu = 0x0, c_cpu = 0x0, dc_cpu = 0x0, binary_input = 0x0, input_layer = 0x0, 
          self_layer = 0x0, output_layer = 0x0, reset_layer = 0x0, update_layer = 0x0, state_layer = 0x0, input_gate_layer = 0x0, state_gate_layer = 0x0, input_save_layer = 0x0, 
          state_save_layer = 0x0, input_state_layer = 0x0, state_state_layer = 0x0, input_z_layer = 0x0, state_z_layer = 0x0, input_r_layer = 0x0, state_r_layer = 0x0, input_h_layer = 0x0, 
          state_h_layer = 0x0, wz = 0x0, uz = 0x1, wr = 0x0, ur = 0x7f01fe5486e0 <stderr>, wh = 0x1, uh = 0x0, uo = 0x2, wo = 0x7f01fe1fd157 <__fprintf+135>, uf = 0x0, wf = 0x3000000018, 
          ui = 0x4000000000000000, wi = 0x7f01fe77bb11 <__ieee754_pow_sse2+785>, ug = 0x0, wg = 0x0, softmax_tree = 0x4008000000000000, workspace_size = 139646535789673}
        delta = {w = 352, h = 448, c = 3, data = 0x556054142a40}
        resized = {w = 352, h = 448, c = 3, data = 0x556054310a50}
        out = {w = 352, h = 448, c = 3, data = 0x5560544dea60}
#3  0x0000556051fdede1 in run_nightmare (argc=<optimized out>, argv=<optimized out>) at ./examples/nightmare.c:382
        layer = 2
        octave = <optimized out>
        buff = "\001\000\000\000\000\000\000\000p1\311\376\001\177", '\000' <repeats 11 times>, "\375\310\376\001\177\000\000\000\000\000\000\000\000\000\000\310\064\311\376\001\177\000\000\240;\377\v\375\177\000\000\200;\377\v\001\000\000\000p1\311\376\001\177\000\000\220;\377\v\375\177\000\000S\254\373Q`U\000\000^\226\223\034\000\000\000\000\377\377\377\377\000\000\000\000\250\321\377\v\375\177\000\000\000\343\033\376\001\177\000\000\220\376\310\376\001\177\000\000\b\000\000\000\000\000\000\000\230\322\377\v\375\177\000\000\b7\311\376\001\177\000\000X<\377\v\375\177\000\000T<\377\v\375\177\000\000\000\000\000\000\000\000\000\000\247\362\060\376\001\177\000\000\360=\377\v\375\177\000\000X<\377\v\375\177\000\000"...
        crop = {w = 1, h = 0, c = 0, data = 0x7ffd0bff3c58}
        resized = {w = 3, h = 0, c = -20380560, data = 0x1}
        cfg = <optimized out>
        weights = <optimized out>
        input = <optimized out>
        range = 1
        norm = 1
        rounds = 1
        iters = 2
        octaves = 4
        zoom = 1
        rate = 0.0399999991
        thresh = 1
        rotate = 0
        momentum = 0.899999976
        lambda = 0.00999999978
        prefix = 0x0
        reconstruct = 0
        smooth_size = 1
        net = {n = 3, batch = 1, seen = 0x556053ee2010, t = 0x556053ee2030, epoch = 0, subdivisions = 1, layers = 0x556053ef19c0, output = 0x7f01ef6ca010, policy = CONSTANT, 
          learning_rate = 0.00999999978, momentum = 0.899999976, decay = 0.000500000024, gamma = 0, scale = 0, power = 4, time_steps = 1, step = 0, max_batches = 0, scales = 0x0, 
          steps = 0x0, num_steps = 0, burn_in = 0, adam = 0, B1 = 0, B2 = 0, eps = 0, inputs = 83916, outputs = 224960, truths = 224960, notruth = 0, h = 189, w = 148, c = 3, max_crop = 20, 
          min_crop = 10, center = 0, angle = 0, aspect = 1, exposure = 1, saturation = 1, hue = 0, gpu_index = -1, hierarchy = 0x0, input = 0x7f01f4d70010, truth = 0x7f01eed28010, 
          delta = 0x556054142a40, workspace = 0x7f01e3fe7010, train = 0, index = 0, cost = 0x556053ee2050}
        cfgbase = 0x556053ee2220 "jnet-conv"
        imbase = 0x556053ee2240 "scream"
        im = {w = 352, h = 448, c = 3, data = 0x7f01f510e010}
        features = 0x0
        update = {w = 0, h = 0, c = -22557532, data = 0x1}
        e = 0
        n = 1
#4  0x0000556051fe2558 in main (argc=8, argv=0x7ffd0bff3ee8) at ./examples/darknet.c:480
No locals.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant