This was associated with line = TRUE, which should be used for a continuous response variable. Since I used nnet for regression (and not for classification), I needed to set linout = T to tell nnet to use linear output
nnet.fit <- nnet(medv/50 ~ ., data=BostonHousing, size=10, linout=TRUE, skip=TRUE, MaxNWts=10000, trace=FALSE, maxit=100)
It worked well for me, hope it helps.
source share